How AI Incident Investigation Works: Two Agents Traced

AI-powered incident investigation is an LLM agent that takes an alert, runs read-only queries against the live system (kubectl, logs, metrics), picks each next query from what the last one returned, and stops when the evidence supports one cause. It then reports that cause with the evidence behind it and a suggested fix that a person approves. It does not replace your monitoring or paging: it does the first pass of an engineer's diagnosis, and it can be wrong in ways you can only catch by reading its trace.

Below are two real investigations of the same fault, one by Aurora and one by HolmesGPT, with every tool call in order. Then the scenario where both agents failed, what happened when their commands were blocked, and what to require from any tool. The faults come from our lab: 19 injected Kubernetes failures, two runs each, all public in Arvo-AI/aurora-lab and summarized on /benchmark. Arvo AI builds Aurora and wrote the scenarios. For a list of the products in this category, see the AI incident investigation tools list; this page is about how the method works.

How does an AI agent investigate an incident?

Every agent we have traced runs the same loop:

1. Read the alert and anything it already knows (past incidents, memory, connected clusters).
2. Look wide: pods, events, logs of the alerted service.
3. Follow the first anomaly in the output, for example an error line in a log.
4. Rule out the obvious suspects (is the dependency down, is the node full).
5. Find the object that explains the anomaly and confirm it with a second piece of evidence.
6. Report: cause, evidence, what it ruled out, a suggested fix.

The model decides each step. Nothing in the loop forces it to look beyond what the alert points at, which is where it fails (see scenario 12 below).

What did Aurora do, call by call?

Scenario 13 of the lab, run r1, 2 October 2026. The fault: a NetworkPolicy named redis allows ingress to redis on 6379 only from pods labeled app=worker. The api pods are not allowed, so every /api/items request times out on its redis call and returns 503. Every pod stays Running and Ready, because the api's /healthz does not touch redis. The alert:

HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%).

Aurora (commit 87d22baf, model Claude Sonnet 5 on Bedrock) had a fresh user and organization, so no memory of earlier runs, and the lab's admin kubeconfig behind its default read-only observability_only command policy. These are all 25 tool calls, start times in UTC, from steps-full.json in the run folder. Indented lines are copied from the tool output, with long lines cut at the right; [...] marks skipped lines, and other bracketed lines are our notes.

20:24:50  list_incidents, grep_memories "frontend", get_connected_clusters
            No matches for 'frontend' in memory.
20:24:54  kubectl get pods -n shop -o wide
            api-f64486c4c-2nqsk         1/1     Running   0          73s
            redis-59458f55d8-n6glm      1/1     Running   0          73s
            [... all 7 pods Running, 0 restarts]
20:24:54  kubectl get events -n shop --sort-by=.lastTimestamp
20:24:54  grep_memories "aurora-lab"
20:25:07  kubectl logs -n shop -l app=frontend --tail=100
20:25:07  kubectl logs -n shop -l app=api --tail=100
            redis connection failed (attempt 1/5): [Errno 111] Connection refused
            connected to redis at redis:6379
            api listening on :8080
            [...]
            request failed: redis error: timed out
            10.244.2.45 "GET /items HTTP/1.0" 503 -
            [... repeats, with /healthz returning 200 in between]
20:25:07  kubectl get configmap -n shop
20:25:25  kubectl logs -n shop -l app=redis --tail=100
            1:M 02 Oct 2026 20:23:45.678 * Ready to accept connections tcp
20:25:25  kubectl describe pod -n shop -l app=redis
20:25:25  kubectl top pods -n shop
            redis-59458f55d8-n6glm      3m           4Mi
20:25:40  kubectl logs -n shop -l app=worker --tail=100
20:25:40  kubectl get configmap api-code -n shop -o yaml
            with socket.create_connection((REDIS_HOST, 6379), timeout=2) as s:
20:25:40  kubectl get svc -n shop
20:26:01  kubectl top nodes
20:26:01  kubectl describe pod -n shop -l app=worker
20:26:01  kubectl get nodes -o wide
20:26:33  kubectl get pod -n shop -l app=loadgen -o yaml
20:26:33  kubectl get networkpolicies -n shop
            NAME    POD-SELECTOR   AGE
            redis   app=redis      2m49s
20:26:33  kubectl get deployment api -n shop -o yaml
20:26:49  kubectl describe networkpolicy redis -n shop
            Created on:   2026-10-02 20:23:54 +0000 UTC
            [...]
              PodSelector:     app=redis
              Allowing ingress traffic:
                To Port: 6379/TCP
                From:
                  PodSelector: app=worker
20:26:49  kubectl get pod -n shop -l app=api --show-labels
            api-f64486c4c-2nqsk   1/1     Running   0          3m8s   app=api,pod-template-hash=f64486c4c
20:27:41  write_memory, again at 20:27:56: "shop namespace frontend 5xx caused by redis NetworkPolicy excluding api pods"

What turned the investigation, in order:

• The api log at 20:25:07. redis error: timed out moved the question from "why is the frontend failing" to "why can the api not reach redis". The first line, "Connection refused", is from startup before redis was up, and the api then connected. The later errors are timeouts, not refusals: the hint that packets are being dropped, not rejected.
• Ruling redis out at 20:25:25. Clean startup log, Ready, 3m CPU. The agent now knows the dependency is up and the path to it is broken.
• The detour at 20:25:40 to 20:26:01. Worker logs, nodes, node usage. None of it was needed. It read the api source from the ConfigMap, which gave it the 2 second client timeout it later cited.
• The policy at 20:26:33. get networkpolicies was call 20 of 25, about 1 minute 40 seconds after the first call. describe showed the app=worker allow-list, and listing the api pods' labels confirmed they carry app=api. Two pieces of evidence, one from each side of the blocked path.

The report's root cause:

The root cause was a NetworkPolicy named redis in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled app=worker only [20, 22]. The api pods are labeled app=api, not app=worker [23], so their connections to redis were silently dropped by the policy [...]

The bracketed numbers point at the tool calls above, so each claim can be checked against its output. The report also had a "Ruled Out" list (redis overload, node contention, startup noise) and a "Not Checked" list. Separately, Aurora suggested a fix with a risk rating:

title:   Patch `redis` NetworkPolicy to allow `app=api` ingress
risk:    low
command: kubectl patch networkpolicy redis -n shop --type='json' -p='[{"op":"add","path":"/spec/ingress/0/from/-","value":{"podSelector":{"matchLabels":{"app":"api"}}}}]'
undo:    null

Nothing ran it; suggestions wait for a person. Note the empty undo: to roll that patch back you would re-apply the original policy manifest. Run total: 202 seconds, 25 tool calls, 38 model calls, $0.69 in LLM cost. Graded correct. The model review still found two overstatements in the report: it called the young pod ages "a full cluster/deployment restart", and it said "100% failure" when one api log showed a 200 before the policy took effect. The NetworkPolicy blocking traffic runbook covers the fix by hand.

What did HolmesGPT do on the same fault?

HolmesGPT 0.43.0, same model, same cluster setup and fault, run on 15 October 2026 with holmes ask and its default toolsets. It got the same alert text with a slightly different suffix: "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." (Aurora's was "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause."). HolmesGPT prints each call as it goes. Its 26 calls, grouped by model turn, from holmes.md in the run folder:

Its conclusion, after 110 seconds:

A NetworkPolicy named redis in namespace shop (created 2026-10-15T09:06:57Z, applied via kubectl apply) restricts ingress to the redis pod (redis-59458f55d8-kspkb, label app=redis) to only allow traffic from pods labeled app=worker, on TCP/6379

It then proposed adding app=api to the policy's from list and listed what it checked and found healthy. Graded correct. Its report says the policy was applied "4s later" than the api's first redis PING, which the judge noted is inferred from timestamps rather than shown.

The two investigations followed the same path: logs of the failing service, the api's redis timeout, redis ruled out, then the policy late in the run (call 20 of 25 for Aurora, 22 of 26 for HolmesGPT). Neither went to NetworkPolicies first, even though a timeout to a healthy pod in the same namespace points there.

Aurora's grades were read by a person at Arvo; HolmesGPT's and K8sGPT's are judge-model only (details below). Aurora r2's 444 seconds includes waiting for a worker slot taken by Aurora's own background discovery jobs in the lab stack. K8sGPT does not take an alert: k8sgpt analyze scans the namespace for objects in a bad state, and with every pod Running and Ready its default analyzers found nothing. That is a different job, not a failed investigation.

Where does AI incident investigation fail?

Scenario 12 is the clearest failure we have. Two unrelated changes went out together: the worker's BATCH_SIZE went from 500 to 2500 (OOMKilled against a 192Mi limit), and the frontend Service's selector was changed to app=frontend,tier=web, which no pod carries. The alert covered only the worker:

KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) is in waiting state (reason: CrashLoopBackOff).

All four agent runs were graded partial. Both agents did good work on the alerted fault (Aurora in both runs and HolmesGPT in r2 worked out the memory per batch from the worker's code) and then stopped. Aurora's version is worse: in both runs it declared the frontend unaffected and cited get pods output as proof (calls 5 and 14 in r1, call 4 in r2), while the Service in front of those pods had no endpoints. A citation shows where a claim came from, not that the evidence supports it. K8sGPT, which scans everything instead of following the alert, saw the second fault in both runs, but its suggested fix was to label the pods by hand with tier=web rather than revert the selector.

The lesson for the method: an investigation is scoped by its alert. "Everything else is healthy" needs a request-path check (Service endpoints, the load generator's or users' errors), not pod status. The runbook for a second fault the alert does not cover is built on these runs, and the Service with no endpoints runbook covers the selector fix.

What happens when the agent's command is blocked?

Both agents run behind a command guard, and both guards blocked reads the agents wanted. All three tools had the same admin kubeconfig in the lab, each behind its own guard.

Aurora's denial reads "Command blocked by organization policy: Mutating kubectl operations". The default policy's deny rule matches the whole rollout verb, so the read-only rollout history and rollout status go down with rollout restart and rollout undo. Our lab fix branch allowed the two read-only forms, and in the 6 runs on that branch rollout history ran 14 times and was never denied. As of 21 October 2026 the default policy on Aurora's main branch still denies the whole verb, and exec with it. HolmesGPT's bash toolset matches commands by prefix against an allow-list; per its docs, the default list includes kubectl get, describe, logs and top but not rollout or exec, and anything else needs approval. Our runs piped the prompt in ("Interactive mode disabled when reading piped input"), so nobody was there to approve and the command did not run.

A blocked read is a hole in the evidence. What separates a usable tool is whether the report says so. Aurora's reports did, in their "Not Checked" lists, though in one earlier run it repeated the policy's wording and called rollout history "a mutating operation" in the report. The scenario 19 miss also shows the cost of a missing fallback: Aurora's own get pods output showed the failing pod was 9 seconds younger than the healthy one, which was enough to see it had loaded the edited ConfigMap. The ConfigMap change not applied to all pods runbook walks through that check.

The trade-off is real: loosening a guard so the agent can run exec also lets it run anything inside your containers. We would allow rollout history and rollout status and keep exec blocked, and accept that some faults then need a person.

Can an AI agent investigate a security incident?

Not the tools and faults on this page. Everything above is operational: a service is failing and the agent finds the change that broke it. A security incident investigation (a suspicious login, a malware alert, data leaving the network) runs on different data, such as SIEM events, endpoint telemetry and identity logs, and is the job of AI security operations (SOC) tools. We have not tested any of those, and the lab has no security scenarios, so we make no claims about them.

Two things do carry over. The loop is the same shape: a request comes in, the agent queries, cites evidence and concludes. And so does the audit question security teams ask about any agent: what exactly did it run? In the lab, Aurora's trace records every command with its full output, including the denied ones (POLICY_DENIED with the command text), and Aurora screens each command against hand-written rules plus a vendored subset of SigmaHQ detection rules before it runs. HolmesGPT prints every tool call and every blocked command in its output. If you are evaluating an investigation agent for your security team's review, ask for that record from a real run, not a description of it.

What should you require from any AI investigation tool?

From the 76 Aurora and HolmesGPT runs in the lab, the requirements that would have caught the problems we saw:

1. Every tool call and its output, in order. You cannot judge an answer without the trace. Both traced agents provide one.
2. Claims tied to specific evidence, and a habit of opening that evidence. Aurora's scenario 12 reports cited real outputs for a false claim.
3. A "ruled out" and a "not checked" list. Not checked is where blocked commands and skipped areas belong.
4. A check beyond the alerted object before anything is declared healthy: endpoints, traffic, the users' side of the request path.
5. A read-only identity with a guard you can read, and a record of what the guard blocked. Decide in advance which blind spots you accept (exec, Secrets, rollout history).
6. Fixes as suggestions with a risk rating and an undo, run only after a person approves. Aurora's scenario 13 suggestion had the rating and no undo.
7. Isolation between trial runs. Aurora keeps memory and an incident index per organization. In our first, superseded batch, all runs shared one organization and Aurora cited "prior incidents" from an earlier fault; after that every run got a fresh organization.
8. Time and cost per run, including background jobs. In the lab, Aurora's per-cluster discovery job cost about twice what the 38 graded investigations did ($29.70 for 30 discovery runs against $14.53).
9. Two runs per fault. HolmesGPT was wrong and then correct on scenario 19; Aurora was partial and then correct on the same fault.

The AI incident investigation tools list has the commands to run these faults against any product in a trial. For one broken pod, the Kubernetes pod diagnoser takes pasted kubectl output and returns the likely cause in your browser.

How accurate is AI incident investigation in our lab?

A judge model (Claude Opus 5.5) graded every answer against the injected fault. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed with every one, and a third pass graded them from scratch, agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A blinded regrade with the tool names removed matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. The two-sided Fisher exact test that /benchmark computes on these totals gives p = 0.52, comparing person-read Aurora grades with judge-only HolmesGPT grades, so read Aurora and HolmesGPT as roughly even on accuracy. HolmesGPT is about three times faster at the median. Aurora's runs are from 2 October 2026, the others from 15 and 16 October, on the same cluster setup (kind v0.33.0, Kubernetes v1.37.0, 3 nodes) and faults. Arvo wrote the scenarios, a kind cluster with one small app is not production, and two runs per fault do not measure variance. The lab report has the method and every miss, and agentic incident response on a real alert traces a third investigation (scenario 07) call by call.

Aurora is open source under Apache 2.0: github.com/Arvo-AI/aurora, or book a demo to see it on your own alerts.

Sources: run folders runs/v1-run1/13-networkpolicy-blocks-redis, runs/v1-run1/12-two-faults-oom-and-selector, runs/v1-run1/19-config-drift-partial-restart and runs/h2h-v1/r1|r2, plus benchmark/results.json, REPORT.md, FINDINGS.md, runs/h2h-v1/README.md, review-summary.md, review2-summary.md and runs/blind-regrade.md in Arvo-AI/aurora-lab. Aurora's default command policy read on its main branch and HolmesGPT's bash toolset docs read on 21 October 2026. Verified 21 October 2026.