Agentic Incident Response: One Real Alert, Every Tool Call

Agentic incident response (also called agentic incident management) is an AI agent that takes an alert, investigates the live system itself by choosing and running read-only commands one after another, and reports a root cause with its evidence and a suggested fix. Nobody scripts its steps in advance, as with runbook automation, and in the tools tested here a person still approves any change it proposes.

Below is one complete agentic investigation from our lab: the alert, every tool call, the root cause, the suggested fix, and the commands the agent was not allowed to run. Then the autonomy levels vendors offer, checked against their docs, and the scenarios where the agents got it wrong. Arvo AI builds Aurora SRE, one of the three tools tested, and wrote the scenarios; every run is public in Arvo-AI/aurora-lab and the live table is at /benchmark. For the wider category, read the AI SRE complete guide; for investigation techniques, AI-powered incident investigation.

What does an agent actually do across the incident lifecycle?

Scenario 07 of the lab, run r1, 2 October 2026. The fault: the api Service's selector was changed from app=api to app=api-v2, a label no pod carries. Every pod stays Running and Ready, nothing restarts, and the frontend's nginx returns 502 on every /api/ request because the Service has no endpoints.

1. The alert. This is the whole input:

HighErrorRate: frontend /api/ requests in namespace shop are failing with 5xx; all pods are Running. Cluster: aurora-lab (kubeconfig). Investigate and find the root cause.

Aurora SRE (commit 87d22baf, model bedrock/us.anthropic.claude-sonnet-5) ran with a fresh user and organization, so no memory from earlier runs, and the lab's admin kubeconfig behind its default observability_only command policy.

2. The investigation. Every tool call, in order, with start times in UTC from steps-full.json in the run folder. Indented lines are copied from the tool output, usually a few lines picked from a longer result; [...] marks a cut inside an excerpt, and any other line in square brackets is our note, not output.

16:36:09  list_incidents, grep_memories
16:36:10  get_connected_clusters
16:36:15  kubectl get pods -n shop -o wide
            api-f64486c4c-4rwgd         1/1     Running   0          79s   10.244.1.12   aurora-lab-2-worker
            api-f64486c4c-56fw6         1/1     Running   0          79s   10.244.2.18   aurora-lab-2-worker2
            [...]
16:36:15  kubectl get events -n shop --sort-by=.lastTimestamp
16:36:15  kubectl get svc,ingress -n shop
            service/api        ClusterIP   10.96.89.40     <none>        8080/TCP   81s
16:36:31  kubectl logs -n shop -l app=frontend --tail=100 --all-containers
            2026/10/02 16:35:10 [error] 29#29: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.1.14 [...]
16:36:31  kubectl logs -n shop -l app=api --tail=100 --all-containers
            api listening on :8080
            10.244.1.1 "GET /healthz HTTP/1.1" 200 -
16:36:31  kubectl logs -n shop -l app=loadgen --tail=100 --all-containers
            2026-10-02T16:35:06Z GET /api/items -> 502
            2026-10-02T16:35:08Z GET /api/items -> 502
            [...]
16:37:05  kubectl get endpoints api -n shop -o yaml
            [...]
            kind: Endpoints
            metadata:
            [...]
              name: api
              namespace: shop
            [...]
            [no subsets field at all]
16:37:05  kubectl logs api-f64486c4c-4rwgd -n shop --tail=200
16:37:05  kubectl logs api-f64486c4c-56fw6 -n shop --tail=200
16:37:20  kubectl get pods -n shop -l app=api -o json
16:37:20  kubectl describe pod api-f64486c4c-4rwgd -n shop
16:37:41  kubectl get endpoints api -n shop
            NAME   ENDPOINTS   AGE
            api    <none>      2m43s
16:37:41  kubectl get svc api -n shop -o yaml
            kubectl.kubernetes.io/last-applied-configuration: |
              {"apiVersion":"v1","kind":"Service", [...] "selector":{"app":"api"}}}
            [...]
            selector:
              app: api-v2
16:37:41  kubectl get endpointslices -n shop -l kubernetes.io/service-name=api -o yaml
            endpoints: null
16:37:56  kubectl get deploy -n shop -o wide
            api        2/2     2            2           2m59s   api          python:3.12-alpine   app=api
16:37:56  kubectl get pods -n shop -l app=api-v2
            (no output)
16:37:56  kubectl get events -n shop --field-selector involvedObject.name=api -o wide
16:38:31  kubectl get nodes -o wide
16:38:31  kubectl top pods -n shop

The first three calls are housekeeping (the incident list, a memory search that found nothing, the cluster list). Then the agent fans out: pods, events, Services, then logs for all three workloads at once. The frontend log gives it connect() failed (111: Connection refused) while connecting to upstream and the loadgen log a 502 every two seconds since 16:35:06 (44 of its 45 lines; one 200 at 16:35:12), so by 16:37:05 it is asking what is behind the api Service, and the Endpoints object comes back with no subsets. At 16:37:41 it has the Service's live selector (app: api-v2) next to its last-applied one, at 16:37:56 the Deployment's label (app=api), and in the same second it tests the alternative, a real api-v2 workload the selector might target on purpose, and gets no output. The last two calls rule out node and resource problems. 22 tool calls (19 of them kubectl), 35 model calls, 175 seconds by Aurora's own timer, $0.52 of model spend.

3. The root cause. The report's paragraph:

The root cause was a label/selector mismatch on the api Service in the shop namespace: the Deployment and its pods are labeled app=api [13, 18], but the Service had no matching backend endpoints, as confirmed by kubectl get endpoints api -n shop returning <none> and the associated EndpointSlice showing endpoints: null [10, 15, 17].

The bracketed numbers are the tool calls above. The report also listed what it ruled out (OOM, crash loops, redis, node capacity, nginx misconfiguration, an intentional api-v2 target) and what it did not check, including when the selector changed and who changed it. Note what the paragraph leaves out: the bad value app=api-v2, which Aurora had read at 16:37:41. The judge model graded the run correct, a second model review called it partial for exactly that omission, and Ben read the full output (the final chat message names app: api-v2) and kept it correct. The second run of the scenario was also correct, in 138 seconds, 15 tool calls and $0.40.

4. The suggested fix. It went into a separate suggestions list, rated low risk: "Patch api Service selector to match pod labels app=api", with the command:

kubectl patch svc api -n shop -p '{"spec":{"selector":{"app":"api"}}}'

5. What it did not do. Aurora did not run the patch, and did not try to: its Not Checked list says mutating kubectl patch operations "were blocked under read-only session policy", which is true of the policy, but no patch call appears in the tool log. The fix is a suggestion a person applies, and the rollback is the same command with app=api-v2, if the new selector turns out to be intended. The step-by-step fix with verification is the Service has no endpoints runbook.

The same alert went to two other open-source tools on 15 October 2026, same cluster, same fault. HolmesGPT 0.43.0, on the same model, got there in 55 seconds and 11 tool calls and read the drift the same way:

The Service's kubectl.kubernetes.io/last-applied-configuration annotation still records selector: {"app":"api"} (the originally-applied/desired state), but the live object's spec.selector is app: api-v2.

K8sGPT 0.4.40 is not an agent; its analyze runs fixed analyzers over the namespace and has a model explain each finding (Claude Sonnet 4.5 here, because K8sGPT sends a temperature setting that Bedrock rejects for Sonnet 5), without reading the alert. It flagged the Service in about 15 seconds, "Service has no endpoints, expected label app=api-v2", then pointed the fix the wrong way: "Add missing label to Pods: kubectl label pods app=api-v2". The judge graded it partial. Flagging the broken object is one step; working out which side of the mismatch is wrong is the part an agent adds.

What was the agent not allowed to run, and why?

All three tools got the same admin kubeconfig, each behind its own guard: Aurora's observability_only command policy, HolmesGPT's command allow-list, K8sGPT's read-only analyze. The guard, not the credential, decided what ran.

Aurora's policy is an organization-level allow-list and deny-list, checked before every command; a new organization gets the observability_only template with both lists on (command_policy.py, unchanged on this point between the lab commit and today's main). Its kubectl allow rule covers only get, describe, logs, top, explain, api-resources, api-versions, cluster-info and read-only config subcommands, so anything else (label, annotate, set image) is refused as "No matching allow rule". On top of that, one deny rule labelled "Mutating kubectl operations" names exec, cp, run, attach, port-forward, apply, delete, create, edit, patch, replace, scale, rollout, drain, cordon, uncordon and taint. Not all of those mutate, but every one is refused with that label. Across all 38 graded Aurora runs (both lab sets, 19 scenarios, 2 runs each) the agent made 452 kubectl calls and the policy refused 30 of them, every one with the same message. None of the 30 was an attempt to change the cluster. In the 38 tool logs there is no patch, apply, delete, scale, edit or label call at all, and the only shell command outside kubectl was one curl to Docker Hub's tag API in scenario 03; the agent investigated and left changes to the suggestions list. What it was refused:

What the agent saw, from scenario 12 r1 and scenario 19 r1:

$ kubectl rollout history deployment/worker -n shop
Command blocked by organization policy: Mutating kubectl operations

$ kubectl exec frontend-76d98597b9-g5zrg -n shop -c frontend -- cat /etc/nginx/conf.d/default.conf
Command blocked by organization policy: Mutating kubectl operations

The exec block is defensible: exec runs arbitrary commands inside a container, and a policy that cannot see what the shell will do should refuse it. The rollout history block was a bug in the default. The deny rule matched the whole rollout verb, so the read-only history and status went down with restart and undo, and the agent lost the first command most people run to answer "what changed". It worked around it by diffing ReplicaSets. A lab branch narrowed the rule to rollout (restart|undo|pause|resume); the fix, which also adds rollout (history|status) to the allow-list, is an open pull request on the Aurora repo and not yet merged. HolmesGPT's own allow-list blocked the same command in 12 of its runs.

Both blocks cost evidence, and the second section below shows where. That is the trade an agentic tool makes: every read it is refused is a question it cannot answer, and every write it is allowed is a change nobody reviewed.

What autonomy levels exist for incident response agents?

Vendors describe autonomy in their own terms. Mapped onto four levels:

How named tools gate actions, from their own docs, checked 21 October 2026:

Of these, only Aurora, HolmesGPT and K8sGPT ran in our lab, and only in read-only, background mode; we did not test Aurora's chat approval prompt, HolmesGPT's remediation toolset or the K8sGPT operator. The rest is what each vendor documents.

Our position: on the lab's evidence, level 2 is where an incident agent belongs in production today. None of the three tools needed write access to be useful, and the failures below are exactly the cases where an agent at level 4 would have acted on a wrong or incomplete answer. Microsoft's own run-modes page recommends Review for production incidents and Autonomous for staging and recurring tasks such as daily health checks; that split is reasonable, as long as the autonomous jobs are ones whose worst action you can undo.

How is agentic incident response different from runbook automation and AIOps?

AIOps reduces the noise that reaches a person; an agent does the first stretch of the on-call engineer's work after the page. Nobody wrote "check the endpoints when nginx says connection refused" for scenario 07; the agent chose it from the log line. The longer comparison is in AI SRE vs AIOps.

How do AI agents handle complex incidents?

Unevenly. Two scenarios were built to catch the failure an agent is prone to: stopping at the first good explanation.

Scenario 12: a second fault the alert did not mention. The worker is OOMKilled after a batch-size change, and separately the frontend Service's selector matches no pods, so users get errors. The alert names only the worker. Aurora found the OOM in both runs and wrote, with pod status as its only evidence:

Other workloads in the shop namespace (api, frontend) remained healthy and unaffected [5, 14].

It never ran get svc or get endpoints in either run, and one of its 18 calls in r1 was the blocked rollout history above. HolmesGPT did the same in both of its runs; its first run ends "Root cause confirmed: shop/worker is being OOMKilled." and makes no claim about the frontend either way. All four runs were graded partial. K8sGPT, which scans every object instead of following the alert, listed the Service in both runs, "Service has no endpoints, expected label tier=web", and never explained the OOM. An agent allowed to act on its own would have raised the worker's memory, closed the incident and left the frontend down. The check that catches it is in the second fault runbook.

Scenario 19: a ConfigMap edit that only one of two frontend pods has loaded. HolmesGPT's first run read the untouched pod's logs and concluded "All data points to a transient startup race, now resolved." Graded wrong. Aurora's first run found the edited upstream (server api:8000 against a Service on 8080) and tied it to the failing pod, then tried to read the live nginx config with kubectl exec, was refused, and wrote that the discrepancy "was not fully reconciled". Its own get pods output already showed the failing pod was 9 seconds younger than the healthy one, which answers the question without exec. Graded partial. Both second runs got it. The manual version is the ConfigMap change not applied to all pods runbook.

Smaller misses run through the whole set. A second model found claims the evidence did not support in 15 of the 18 Aurora reports from the harder scenario set, none of them in the root cause: "both cluster nodes" when there are three, "pulled successfully" with no Pulled event.

The totals, from results.json:

Aurora's 38 graded runs are from 2 October 2026 and HolmesGPT's and K8sGPT's from 15 and 16 October, on the same cluster and faults. Each agent got the alert text; Aurora's prompt added "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." and HolmesGPT's "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." Aurora's 38 grades were read by a person, who changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model review agreed with all 76, a third pass graded them from scratch and disputed 6 K8sGPT labels, and a blinded regrade with the agent names removed matched 38 of 38 Aurora, 37 of 38 HolmesGPT and 36 of 38 K8sGPT verdicts, every change downward. On correct against not, the gap between Aurora and HolmesGPT is not statistically significant (two-sided Fisher exact p = 0.52, computed on /benchmark from these totals, comparing person-read Aurora grades with judge-only HolmesGPT grades). HolmesGPT is about three times faster at the median. Read the two as roughly even on these faults.

What should you expect from an agent on your own incidents?

1. Start at level 1 or 2. A read-only identity such as the Kubernetes view ClusterRole reproduces most of scenario 07. Decide in advance which reads you refuse (Secrets, exec) and check that the policy does not refuse read-only commands like rollout history by accident.
2. Grade the answers against what the human found. Correct, partial or wrong, the same day, including false claims outside the root cause. That record is what tells you whether to move a class of incident up a level.
3. Treat "everything else is healthy" as a claim that needs a request-path check. Pod status is not evidence that a Service has endpoints. In scenario 12 Aurora said it in both runs ("remained healthy and unaffected", "running normally and unaffected") and it was wrong both times; HolmesGPT never checked.
4. Keep changes behind approval, with an undo. A suggested command with a risk rating and a rollback, or a pull request someone merges.

The pilot procedure step by step is in the AI SRE complete guide, and the tool landscape in top AI SRE tools 2026. The faults are reproducible: ./lab.sh break 07-service-selector-mismatch in the lab repo puts a cluster in the state above. To check a symptom you are seeing now, paste it into /diagnose.

Aurora SRE is open source under Apache 2.0: github.com/Arvo-AI/aurora, or book a demo.

Sources: Aurora run folders runs/v0-run2/07-service-selector-mismatch/r1 and r2, runs/v1-run1/12-two-faults-oom-and-selector, runs/v1-run1/16-dns-search-domain, runs/v1-run1/19-config-drift-partial-restart and the steps-full.json tool logs of all 38 graded runs; HolmesGPT and K8sGPT outputs under runs/h2h-v1/r1 and r2; benchmark/results.json, REPORT.md, FINDINGS.md, runs/h2h-v1/README.md, review-summary.md, review2-summary.md and runs/blind-regrade.md in Arvo-AI/aurora-lab. Vendor autonomy and approval behaviour from the linked docs of HolmesGPT, the K8sGPT operator, Azure SRE Agent, Resolve AI and Cleric, read 21 October 2026. Verified 21 October 2026.