AI Root Cause Analysis vs 5 Whys for SREs: 3 Tools Tested

AI root cause analysis is an agent that takes the alert, runs read-only queries against the cluster and the logs, and returns a cited hypothesis in minutes. The 5 Whys is a team method that follows one causal chain from the symptom down to a condition it can remove; it works when the incident has one chain and fails when it has several. Neither one proves causation on its own, and they fail the same way: both stop at the first chain that explains the alert. In our lab one AI agent named the right cause in 34 of 38 runs; in three of its 4 partial answers it explained the alerted fault and never checked a second change, and in the fourth it had the evidence but stopped one proof short.

The evidence below comes from the public Aurora lab: 19 faults injected one at a time into a small Kubernetes app, each investigated twice from the alert text alone, with every tool call and report published. Arvo builds Aurora and wrote the scenarios, so read the Aurora numbers with that in mind.

Which RCA method should an SRE use, and when does each fail?

Use a timeline on every incident, change correlation first inside it, the 5 Whys when one chain explains the impact, and a fault tree when the failure needed several conditions at once. An AI agent speeds up the evidence for all four but inherits the same blind spots.

What is root cause analysis in SRE?

Root cause analysis (RCA) is the investigation of an incident that finds the conditions that caused it, so the team removes those conditions instead of only the visible symptom. In Google's model it is the core of the postmortem, which the SRE book defines as "a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring."

The SRE Workbook separates two things worth naming apart: "Identifying the root causes and trigger is one of the most important reasons to write a postmortem." The trigger is the event that set the failure off, such as a config push. The root causes are the latent conditions that let the trigger become an outage, such as a probe that cannot see the broken path. A postmortem that names only the trigger fixes one instance and leaves the conditions in place.

RCA is also not a search for someone to blame. The SRE book says a blameless postmortem "must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior."

Is there always a single root cause?

Usually not. John Allspaw writes in Each necessary, but only jointly sufficient that for complex systems "there is no root cause," and that failures take "multiple contributing causes, each necessary but only jointly sufficient." Richard Cook's How Complex Systems Fail is blunter: "Post-accident attribution to a 'root cause' is fundamentally wrong."

The term survives because a team needs somewhere to start. The working rule: keep asking after the first cause is found, and record every condition that had to be true. Scenario 19 below is a small, concrete case of four conditions, each necessary.

5 Whys vs AI root cause analysis: where does each break?

The Lean Enterprise Institute defines the method as "the practice of asking why repeatedly whenever a problem is encountered in order to get beyond the obvious symptoms to discover the root cause," and adds that "The specific number five is not the point." Stop when the answer is something you can eliminate, not when you reach five.

Its weakness is that it follows one chain. Lab scenario 12 shows the failure mode cleanly, and an AI agent fell into it too.

The setup (scenario 12). Two unrelated changes went out back to back:

kubectl -n shop set env deploy/worker BATCH_SIZE=2500
kubectl -n shop patch svc frontend -p '{"spec":{"selector":{"app":"frontend","tier":"web"}}}'

The alert covered only the first one: "KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) is in waiting state (reason: CrashLoopBackOff)." The second change left the frontend Service with no endpoints, so every user request failed.

A 5 Whys from the alert lands exactly where Aurora's report landed:

1. Why is the worker crash looping? It is OOMKilled (exit code 137) under a 192Mi limit.
2. Why does it run out of memory? BATCH_SIZE went from 500 to 2500. Aurora read the worker code and did the math: about 100 KiB per item, so about 244 MiB at 2500.
3. Why did that ship? The env var changed without the memory limit changing.

Every step is right, and the chain never reaches the frontend. The load generator's log shows that users were failing first: last success at 20:23:32Z, first timeout at 20:23:35Z, while the incident was created from the worker alert at 20:24:25, about 50 s later.

What Aurora did (runs). In both runs it found the OOM and the BATCH_SIZE change and stopped there. Run r1's report says "Other workloads in the shop namespace (api, frontend) remained healthy and unaffected," and its only evidence was pod status. It never ran get svc, get endpoints or read the load generator's log. Its suggested fix reverted BATCH_SIZE and left the selector broken. A person reviewed both runs: partial.

It was not alone. HolmesGPT 0.43.0, on the same model, also diagnosed only the OOM in both runs. K8sGPT 0.4.40, which is not given the alert and scans every object in the namespace, flagged Service has no endpoints, expected label tier=web in both runs. It also listed the OOMKilled worker, but never said why it ran out of memory: its advice was a larger memory limit, and for the Service it suggested adding tier=web to the pods rather than reverting the selector. A scanner sees breadth that an alert-driven investigator does not, and explains less.

The lesson for humans and agents alike. Before writing "unaffected", check the user path: the load generator or synthetic check, the Service endpoints, the error rate at the edge. A pod in Running says nothing about whether traffic reaches it. The full procedure is in the runbook second fault when the alert covers only one.

When is a fault tree better than the 5 Whys?

When the outage needed several things to be true at once. Fault tree analysis starts from the undesired event and decomposes it with AND gates (all inputs needed) and OR gates (any one is enough). The U.S. Nuclear Regulatory Commission's Fault Tree Handbook (NUREG-0492), published January 1981, is still the standard reference. A related group tool, the fishbone diagram, is described by the Lean Enterprise Institute as "a tool used to identify the root causes of a problem"; it generates candidates, while a fault tree states how they combine.

Lab scenario 19 is a small fault tree. The alert: "HighErrorRate [warning] namespace=shop service=frontend: 48% of requests to /api/ returned 5xx over the last 10 minutes (threshold 5%)." What we had done: edited the frontend-nginx ConfigMap upstream from api:8080 to api:8000, then deleted one of the two frontend pods.

About half of /api/ requests return 5xx
└── AND
    ├── ConfigMap upstream edited to api:8000 (the api Service exposes only 8080)
    ├── one of two frontend pods recreated after the edit
    ├── nginx reads its config only at start, so the old pod keeps api:8080 in memory
    └── the readiness probe hits /healthz, which nginx answers itself,
        so the broken pod stays Ready and keeps receiving traffic

Remove any one branch and the incident does not happen, which is also the list of action items: validate upstreams against Services before applying, restart all replicas on a config change, and probe a path that exercises the upstream. A 5 Whys from "requests fail" tends to stop at the first branch. The trap the tree exposes: restarting the "healthy" pod without fixing the ConfigMap would break it too.

How do you do RCA for distributed systems?

Build one timeline across every component, then test the change that came first. Google's troubleshooting chapter explains why the timeline so often points at a change: "Systems have inertia," and "Recent changes to a system can be a productive place to start identifying what's going wrong." It also names the trap: "Hunting down spurious correlations that are actually coincidences or are correlated with shared causes."

Scenario 19's timeline, from Aurora's run r2 tool output (2026-10-02, UTC):

Two things in that table are easy to get wrong. The 502s at 21:07:54 are real errors on the pod that later turned out healthy: a cold start, not the incident. HolmesGPT's first run on this scenario blamed exactly that, "a transient cold-start race in which api pods were not yet Ready" in the judge's words, declared the incident resolved, and was graded wrong. Its second run was graded correct, though the judge noted it got the mechanism wrong (it predicted the healthy pod would break on the next kubelet sync), and the blinded regrade called it partial. And the ConfigMap edit itself appears nowhere in the namespace events (52 lines in the evidence captured before the run), and it creates no Deployment revision: the Deployment docs say a new revision is created "if and only if the Deployment's Pod template" changes.

That is why distributed RCA is mostly evidence work:

• Evidence expires. The Kubernetes logging docs say "By default, if a container restarts, the kubelet keeps one terminated container with its logs," and an evicted pod's containers go "along with their logs." Use kubectl logs --previous early and save the output. In scenario 12 r1, Aurora's --previous call failed with a containerd error, and it said so under "Not checked".
• Compare replicas, not services. Scenario 19 is invisible at the Deployment level: every pod 1/1 Running, zero restarts, same revision. It shows only when you read each pod's log and compare the upstream address.
• Use pod age as change evidence. kubectl get pods -o wide showed r4thz at 59 s and rr7xl at 68 s. A replica younger than its siblings, with no rollout, means something replaced it: a delete, an eviction or a node failure.

What does AI-assisted root cause analysis actually do?

It runs the timeline and correlation work above as a loop of read-only commands, then writes a report that cites them. Here is Aurora's complete evidence chain on scenario 19, run r2: 28 tool calls, 253 s, $0.84, graded correct by the judge and by a person (run folder).

The line that closed it, from step 24:

2026/10/02 21:08:10 [error] 27#27: *3 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.244.1.54, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.137.47:8000/items", host: "frontend"

The report then put the branches of the fault tree together: the ConfigMap points at 8000, the Service exposes 8080, nginx does not hot-reload, so the pod started after the edit fails and the one started before it does not. It listed what it ruled out (resource exhaustion, the worker and redis, the api pods) and, under "Not checked", the two commands its policy blocked. Its suggested fix patched the ConfigMap back to api:8080 and then ran kubectl rollout restart deployment/frontend. Aurora suggests; a person runs it. A rollout restart replaces every frontend pod, so apply the ConfigMap fix first, or the restart moves the failure from half the pods to all of them.

Compared with a person, the agent's advantages here were breadth and patience: it read the logs of six of the seven pods, including the load generator, which people rarely think to check. The blocked exec cost less than it looks. The ConfigMap docs say that when a mounted ConfigMap is updated, "projected keys are eventually updated as well", so after a kubelet sync the file in the old pod can already read api:8000 while nginx still runs the api:8080 config it loaded at start. Reading the file proves what is on disk, not what the process loaded. Pod age and per-pod logs were the proof that mattered, and Aurora had both.

Where was the AI only partly right, and why?

On the same fault, run r1 (280 s, 15 tool calls, $0.39) was graded partial by the judge, a second model and a person. It had the same key evidence: the ConfigMap says api:8000, the Service says 8080, one pod times out against 8000, the other recovered against 8080. It wrote "The most likely cause is a port mismatch" and then: "This discrepancy was not fully reconciled", because kubectl exec to read the live config was blocked. It never concluded that the ConfigMap had been edited and only the recreated pod had loaded it. It also left the api backend open as a possible contributing factor although it had not read the api logs.

The evidence to close the gap was already in its output: the 9 s age difference between the pods, and a ConfigMap whose live resourceVersion differed from the one recorded in its last-applied-configuration annotation. When the check it reached for (reading the file inside the pod) was denied, it did not fall back to those, although they were the stronger evidence anyway. The lab's FINDINGS.md records this as a gap, along with the policy behavior behind it: Aurora's default read-only policy denies kubectl exec, which is defensible, and also the read-only rollout history, which is a bug the lab patched on a fix branch.

Two lessons carry over to any AI RCA tool:

1. Hedging is not the failure; stopping is. "Not fully reconciled" was accurate. The miss was not trying the next-cheapest proof.
2. Permissions shape the answer. An agent denied the change history will reason around it, sometimes well (r2) and sometimes not (r1). Read the "not checked" list before trusting the conclusion.

The full diagnosis procedure for this failure, with per-pod commands and the fix, is in the runbook ConfigMap change not applied to all pods.

Which automated root cause analysis tools did we test, and how accurate were they?

We ran three open-source tools an SRE team can install today, Aurora, HolmesGPT and K8sGPT, on 19 injected Kubernetes faults, two runs each, from results.json (full table on the AI SRE benchmark page):

What the table does and does not show:

• Review status differs. Aurora's 38 grades were read by a person. The 76 HolmesGPT and K8sGPT grades are judge-model only (Claude Opus 5.5). A second model read every one of them next to the judge's grade and agreed with all 76. A third pass graded all 76 from scratch without seeing the judge's grade; it agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A blinded regrade with tool names removed matched the published verdicts in 38 of 38 for Aurora, 37 of 38 for HolmesGPT and 36 of 38 for K8sGPT. No person has checked the HolmesGPT or K8sGPT grades yet.
• HolmesGPT is about 3x faster at the median. That is a real advantage during an incident. Aurora's times also include queueing behind background jobs in the lab, described in the lab report.
• K8sGPT does a different job. k8sgpt analyze --explain --namespace shop lists objects that look unhealthy and explains each; it gets no alert and does not investigate one. It printed "No problems detected" on 6 of the 19 scenarios (12 runs), and it was the only tool that flagged the second fault in scenario 12.
• Prompts differed slightly. HolmesGPT got the alert text plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." Aurora got the alert text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." All three got the same admin kubeconfig, each behind its own guard: Aurora's observability_only command policy, HolmesGPT's allow-list, and K8sGPT's read-only analyze.
• Dates and versions. Aurora's graded runs are from 2026-10-02; the HolmesGPT and K8sGPT runs from 2026-10-15 and 16, on the same cluster and faults. HolmesGPT 0.44.0 and 0.45.0 were already released when we ran 0.43.0; we have not rerun on them.
• It is a lab, not production. 19 known faults in a four-service app, written by the team that builds one of the tools. The value is that every run is public and the method can be rerun against any tool you trial.

Independent research points the same way. In the OpenRCA 2.0 preprint (500 instances, 11 frontier LLMs), models found at least one correct root-cause service in 76.0% of cases, grounded it in a verified causal path in only 61.5%, and recovered the exact root-cause set in 20.7%. The authors call the gap "the ungrounded diagnosis." Naming the right component is easier than proving why it failed, which is what scenario 19 r1 looks like from the inside.

Why isn't my AI SRE tool finding the real root cause?

From what we watched across 38 Aurora runs and 76 runs of the other two tools, the causes were rarely the model's reasoning. They were inputs and scope:

• The alert sets the scope. Both agents explained the alerted fault in scenario 12 and missed the unalerted one. Give the agent the user-facing symptom (error rate, synthetic check), not only the component alert.
• Permissions remove the evidence. rollout history was denied in 26 of 38 Aurora runs by its own default policy. Check what the agent's credentials can read, especially change history, before blaming the model.
• No traffic signal, no proof of health. In scenario 19 the load generator log showed the 200 and 504 responses interleaving; without a request log, metrics or traces, "healthy" means "Running".
• Config that is not in a revision. ConfigMap edits, feature flags and console changes leave no rollout. If the agent cannot read the audit log or GitOps history, it has to infer the change from side effects like pod age.
• Unsupported side claims. Several reports carried small errors outside the root cause, such as "both cluster nodes" when the cluster had three. Treat the citations as the product and the prose as a summary.

How should an SRE run an RCA, step by step?

1. Stop the bleeding. The SRE book: "Stopping the bleeding should be your first priority; you aren't helping your users if the system dies while you're root-causing."
2. Preserve evidence. Save --previous logs, per-pod logs, the live manifests and the exact versions before restarts and autoscaling erase them. Rapid triage, the same chapter says, "doesn't preclude taking steps to preserve evidence."
3. Build one timeline. Deploys, config edits, alerts, per-pod log lines and human actions on one clock, with the user-facing onset marked separately from the alert.
4. Correlate with changes, then test. Start with what changed, including changes that leave no revision. Treat a match in time as a hypothesis and test it, for example by comparing a pod that got the change with one that did not.
5. Widen to every condition. Use a fault tree or several 5 Whys branches until every necessary condition is listed, including why detection was late or pointed at the wrong component.
6. Write preventive action items. The SRE Workbook warns about a postmortem whose action items "are mostly mitigative": "you should include some preventative action items and fixes."
7. Write it up blamelessly. Assume people acted reasonably on what they knew. The draft can be generated; see automated post-mortem generation.

An AI agent can do most of steps 2 to 4 in minutes. Steps 5 and 6, deciding which conditions to remove, are where a team's judgment matters most.

How do you automate post-incident reviews with 5 Whys?

Let the agent produce the evidence chain and the first whys, then run the remaining whys as a team on the conditions it surfaced. In scenario 19 the agent's report answers "why did half the requests fail" with citations. The useful human whys come after: why could a ConfigMap point at a port no Service exposes, why did the readiness probe not notice, why did only one pod restart. Each answer is an action item. For a single pod failure, the Kubernetes pod diagnoser takes pasted kubectl output and returns the likely cause, a quick first step before the review.

How does Aurora approach root cause analysis?

Aurora is an open-source (Apache 2.0) AI SRE agent. It takes an alert, runs read-only commands against Kubernetes and the clouds it is connected to (AWS, Azure, GCP, OVH, Scaleway, Fly.io), and writes a report with numbered citations, a "Ruled out" list and a "Not checked" list, as in the runs above. Fixes go into a separate suggestions list; it can open a remediation pull request on GitHub for a person to review, with no auto-merge. Its alert correlation engine links related alerts into the running investigation and never suppresses them. The lab shows where it still falls short: scope set by the alert, and fallbacks when a command is denied.

To try it on your own incidents, the code is on GitHub, or book a demo.

Related: Aurora lab report on 19 Kubernetes failures, HolmesGPT vs K8sGPT, open source incident management.

Sources and verification. Lab data: Arvo-AI/aurora-lab, scenarios 12 and 19, runs v1-run1 (Aurora, 2026-10-02) and h2h-v1 (HolmesGPT 0.43.0 and K8sGPT 0.4.40, 2026-10-15 and 16), kind v0.33.0, Kubernetes v1.37.0, nginx 1.27.5; totals from benchmark/results.json generated 2026-10-21. Quotes from Google's SRE book chapters "Postmortem Culture" and "Effective Troubleshooting", the SRE Workbook chapter "Postmortem Culture", the Lean Enterprise Institute lexicon, John Allspaw's kitchensoap.com essay, Richard Cook's How Complex Systems Fail, the NRC NUREG-0492 page, the Kubernetes logging, ConfigMap and Deployment docs, the HolmesGPT GitHub releases page, and arXiv:2606.27154 (a preprint). All checked against the live pages on 21 October 2026.