AI SRE in 2026: What It Is, Tested on 19 Kubernetes Faults

An AI SRE is a software agent that takes an alert, investigates the live system with read access to tools like kubectl, logs and metrics, and returns a root cause with the evidence behind it. It does the first hour of on-call triage; a person still decides what to change. In our lab, Aurora found the injected root cause in 34 of 38 runs (89%, median 154 seconds) and HolmesGPT in 31 of 38 (82%, median 45 seconds, graded by a judge model rather than a person), and both missed a second outage the alert did not mention.

Everything below comes from that lab: 19 Kubernetes faults we injected on purpose, three agents, two runs each, every transcript public in Arvo-AI/aurora-lab. Arvo builds Aurora and wrote the scenarios, so read the numbers with that in mind. The live table is at /benchmark.

What does an AI SRE actually do during an incident?

Here is one complete Aurora investigation, scenario 15 of the lab, run r1 on 2 October 2026. The fault: a second Deployment, shop-api, an old build missing its DATABASE_URL setting, was applied with the pod label app=api. The api Service selects on app=api, so it load-balances over three pods, one of which answers every real request with a 500 while passing its health check. Every pod shows Running and 1/1 ready.

Aurora received only this text, the scenario's alert plus the lab's one-line instruction:

HighErrorRate [warning] namespace=shop service=frontend: 31% of requests to /api/ returned 5xx over the last 10 minutes (threshold 5%). Cluster: aurora-lab (kubeconfig). Investigate and find the root cause.

The tool calls it made, in order (start times UTC, from steps-full.json in the run folder). Indented lines are copied from the tool output; [...] marks lines we cut:

20:29:58  list_incidents, grep_memories, get_connected_clusters
20:30:03  kubectl get pods -n shop -o wide
20:30:03  kubectl get events -n shop --sort-by=.lastTimestamp
20:30:15  kubectl get svc -n shop
20:30:15  kubectl logs frontend-76d98597b9-bb4z9 -n shop --tail=100
20:30:26  kubectl logs api-f64486c4c-r95bj -n shop --tail=100
20:30:26  kubectl logs api-f64486c4c-vftsb -n shop --tail=100
20:30:26  kubectl get configmap api-code -n shop -o yaml
20:30:43  kubectl get endpoints api -n shop -o wide
            NAME   ENDPOINTS                                            AGE
            api    10.244.1.37:8080,10.244.1.40:8080,10.244.2.48:8080   111s
20:30:43  kubectl get pods -n shop --show-labels
            [...]
            shop-api-69f6fc5964-jhzz8   1/1     Running   0          105s   app.kubernetes.io/name=shop-api,app=api,pod-template-hash=69f6fc5964
            [...]
20:30:43  kubectl logs shop-api-69f6fc5964-jhzz8 -n shop --tail=100
            10.244.1.38 "GET /items HTTP/1.0" 500 -
            [...]
            KeyError: 'DATABASE_URL'
            10.244.1.1 "GET /healthz HTTP/1.1" 200 -
20:31:02  kubectl get deployment shop-api -n shop -o yaml
20:31:02  kubectl get svc api -n shop -o yaml
20:31:02  kubectl get deployment api -n shop -o yaml
20:31:28  kubectl get secrets -n shop
20:31:28  kubectl get configmap shop-api -n shop -o yaml
20:32:03  kubectl get nodes -o wide
20:32:03  kubectl top pods -n shop
20:32:40  write_memory

The excerpt leaves out a few calls (a get configmap list, a list_services call and one failed jsonpath query that came back Error from server (NotFound) because of an unquoted space). The run made 24 tool calls and 36 model calls in total: 10 investigation steps, the report and the fix suggestions on Claude Sonnet 5, and 23 command-safety checks plus one summary on Claude Haiku 4.5.

The turn happens at 20:30:43: three endpoints behind a Service whose Deployment runs two replicas, then a pod whose labels include app=api but whose name is not api-, then its logs. The report's root cause ends:

With 1 of the 3 load-balanced endpoints behind api failing 100% of the time, roughly one-third of frontend /api/ requests fail, matching the alert's reported error rate [8, 13, 15].

It also listed what it ruled out (redis, node resources, nginx config) and what it did not check, including how the colliding Deployment got applied. Its first suggested next step was "Scale shop-api Deployment to zero replicas", rated low risk; the second, relabelling the Deployment, was rated high risk and came with an undo command. Aurora ran neither: its command policy was read-only, and a person would have to.

Cost and time: 175 seconds by Aurora's own timer and $0.59 of model spend for all 36 calls. The second run of the same scenario took 118 seconds and $0.35. HolmesGPT also got this one right in both runs (97 and 162 seconds). K8sGPT printed "No problems detected", which is accurate for what it checks: every object was healthy. The manual version of this diagnosis is in the runbook Service selects the wrong pods.

That is the job in practice. Some products add scheduled checks on top (HolmesGPT's operator mode, Azure SRE Agent's scheduled tasks), but the part you can grade is this loop: read an alert, form a guess, run commands to test it, and write down what it found and what it did not look at.

How accurate are AI SRE agents today?

We ran Aurora, HolmesGPT 0.43.0 and K8sGPT 0.4.40 on the same 19 faults on a kind cluster (Kubernetes v1.37.0, 3 nodes), two runs per scenario, 38 runs per tool. A judge model (Claude Opus 5.5) graded each answer against the injected fault: correct means every required finding is present and none of the listed wrong answers is asserted.

What the table supports:

• Aurora and HolmesGPT are roughly even on accuracy. Three more correct runs out of 38 is not a significant gap: a two-sided Fisher exact test on the totals, the one /benchmark computes, gives p = 0.52. That test also compares person-read Aurora grades with judge-only HolmesGPT grades.
• HolmesGPT is about 3x faster at the median, 45 seconds against 153.5. Part of Aurora's time is contention from its own background jobs in the lab stack, but not all of it.
• HolmesGPT's and K8sGPT's 76 grades have not been checked by a person. A second model reviewed them and agreed with all 76. A third pass graded them from scratch, agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A blinded regrade, with agent and tool names removed, matched the published verdict for 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs; every change was a downgrade.
• K8sGPT does a different job. analyze scans a namespace and explains objects its analyzers flag. It never sees the alert, so a criterion built for alert-driven investigation is not a fair test of it. Its 12 "No problems detected" runs are both runs of six scenarios where its analyzers flagged nothing, scenario 15 among them. The flip side shows up in scenario 12 below: because it lists every problem it detects, it flagged the outage both investigating agents missed.

The prompts were not identical. Aurora's alert text ended with "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause."; HolmesGPT's ended with "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." All three got the same cluster admin kubeconfig, each behind its own guard: Aurora's observability_only command policy, HolmesGPT's allow-list of known commands, and K8sGPT's read-only analyze. No transcript shows a mutating command running.

Where do AI SRE agents fail?

The misses say more than the score. All of them are in the repo with full transcripts.

They stop at the alerted fault. Scenario 12 has two faults: a worker OOM, which the alert names, and a frontend Service selecting no pods, which it does not. Aurora and HolmesGPT found the OOM in all four runs and missed the second outage in all four. Aurora never ran get svc or get endpoints, and neither agent's answer mentions the frontend Service. K8sGPT, which scans everything, listed "Service has no endpoints, expected label tier=web" for shop/frontend in both runs, though it never named the BATCH_SIZE change behind the OOM. Aurora then wrote "Other workloads in the shop namespace (api, frontend) remained healthy and unaffected", with pod status as its only evidence. Treat any "everything else is healthy" line as unverified unless the agent checked traffic. The runbook for this case: a second fault the alert does not cover.

They settle on a plausible story. In scenario 19 one of two frontend pods had loaded an edited ConfigMap and the other had not. HolmesGPT's first run concluded "All data points to a transient startup race, now resolved." and was graded wrong. Aurora's first run found the bad upstream (api:8000) in the ConfigMap but did not work out that only the recreated pod had loaded it. It tried to read the live config with kubectl exec, its read-only policy blocked that, and it left the question open. The clue was in its own get pods output: the failing pod was 9 seconds younger. Both second runs were graded correct, though the blinded regrade marked HolmesGPT's second run partial for getting the mechanism wrong. Runbook: ConfigMap change not applied to all pods.

They skip the red herring. Scenario 11 puts noisy but harmless errors in the worker's logs next to the real fault, a wrong Service targetPort. Both agents found the port. In three of the four runs the agent never ruled out the worker's errors (two never mentioned the worker; one only called it "unaffected"), so those runs were graded partial.

They add small false details. A second-model review found claims the evidence does not support in 15 of Aurora's 18 reports on the harder scenarios (11 to 19), none of them in the root cause: "both cluster nodes" when there are three, an image "pulled successfully" with no Pulled event, a trend drawn from one sample.

Read-only policies cost information. Aurora's default policy blocked the read-only kubectl rollout history in 26 of 38 runs (27 calls), with "Command blocked by organization policy: Mutating kubectl operations", because the deny rule matched the whole rollout verb. In 19 of those 26 runs Aurora queried ReplicaSets instead, which works, but "what changed" is the first question in most incidents.

Are AI SRE agents production-ready in 2026?

For read-only investigation with a person reading the answer: yes, on faults like these. Two independent agents got the full root cause in over 80% of runs. For unattended action: no. Both investigating agents missed an unalerted outage that a plain K8sGPT scan listed, and one declared an incident "resolved" that was not. An agent whose suggested fix is right 34 times in 38 (Aurora's rate on its "Suggested Next Steps" list) is useful and is also wrong often enough that a person must approve each change.

The lab is a small app on a 3-node kind cluster with no real traffic and no history. Production is noisier, which is harder for an agent, and it has dashboards and runbooks the agent can use. These numbers say what these tools can do on Kubernetes faults with known causes; they do not predict your mean time to recovery.

How do you pilot an AI SRE without disrupting production?

Run it the way the lab does: read-only access, real alerts, and a grade for every answer.

1. Give it read-only credentials first. In Kubernetes, the built-in view ClusterRole "allows read-only access to see most objects in a namespace" and does not allow viewing Secrets (Kubernetes RBAC docs). Expect gaps: Aurora's trace above lists Secrets, which view would refuse, and Aurora's own default policy blocked exec and rollout history in our runs. Decide which gaps you accept before the pilot, not during an incident.
2. Send it alerts that already fire. One service, one alert source. The agent investigates in parallel with the on-call engineer and changes nothing.
3. Grade every answer against what the human found. Correct, partial or wrong, plus whether it claimed anything false. Write the grade down the same day.
4. Watch the background cost, not only the per-investigation cost. In our lab, Aurora's discovery job, which runs once per cluster connection, cost about $1 a run: $30.38 that day (30 completed runs plus a few cut short), more than twice the $14.53 for all 38 graded investigations.
5. Allow suggested changes only behind approval. Aurora suggests fixes and can open a pull request, but a person approves and merges; it does not auto-remediate. Check what your tool does by default. Azure SRE Agent, for example, has Review and Autonomous run modes. The agent-level setting defaults to Review, but the run-modes page lists Autonomous as the default for incident response plans and scheduled tasks, and Review only gates Azure infrastructure operations, so set it per plan (overview, run modes).

How long a pilot takes depends on how many alerts you can grade, not on calendar time. One number from our lab to calibrate: 38 graded runs per agent were not enough to tell two agents three points apart (p = 0.52, above). If you are choosing between tools, plan for more answers than that, or accept that only large gaps will show. Microsoft's run-modes page suggests watching Review-mode recommendations for two to four weeks before switching the triggers you consistently approve to Autonomous.

How do you evaluate an AI SRE platform?

Break something on purpose and score the answer. A known fault is the only way to know whether a root cause is right; real incidents rarely come with ground truth. The lab is built for this and is Apache 2.0:

git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab
./lab.sh up && ./lab.sh break 03-bad-image-tag
./runner/h2h.sh 03-bad-image-tag runs/mine

h2h.sh runs HolmesGPT and K8sGPT against the broken cluster and saves their output; it expects both installed and AWS Bedrock access for the model.

Each scenario folder holds the fault (break.sh), proof it is visible (check.sh) and a scenario.yaml with the alert, the findings a correct answer must contain and the wrong answers it must not assert. To put another agent on the public board, open an issue with a script that takes a kubeconfig and an alert and prints a root cause; we run it on all 19 faults and publish the result.

What we learned about grading while building it:

• Grade everything the tool shows. Our first report said 28 of 38 Aurora runs proposed no fix. Wrong: the grader read only the RCA text, and Aurora puts fixes in a separate list.
• Run each scenario at least twice. Scenario 19 split correct and wrong between runs for HolmesGPT and correct and partial for Aurora.
• Include faults the alert does not name and noise next to the real fault. Those two scenarios (11 and 12) account for three of Aurora's four partial runs and four of HolmesGPT's six.
• Blind the grader. Remove agent and tool names before a model grades. It moved 3 of 114 verdicts here, all down.
• Record time and cost per run, including jobs that run outside the investigation.

A longer checklist covering trust, data handling and total cost is in how to evaluate an AI SRE platform. The accuracy question on its own is covered in AI SRE accuracy benchmarks.

Can AI SRE platforms run on-prem or with bring-your-own-model?

Some can. What each vendor says, checked on 21 October 2026:

Two cautions. First, self-hosting the agent does not keep data in your network if the model is a hosted API: prompts and tool output go to the model provider. Only a local model (Ollama, LocalAI, an in-VPC gateway) keeps it inside. Second, all our lab results used Claude models on Bedrock. We have not tested any agent on a local model, and accuracy will not carry over automatically. The architecture options are covered in self-hosted AI SRE.

What is the ROI of an AI SRE for a large engineering organization?

The per-investigation model cost is small next to an engineer's time: Aurora's median was $0.35 per run in our lab, with a range of $0.19 to $0.84, and $14.53 for all 38. Commercial prices include the product, not only the model, and some are published: Cleric lists "$5 to triage and address an alert" (10 credits at $0.50), with plans from $700 a month (pricing). Azure SRE Agent bills an always-on charge of 4 Azure Agent Units an hour per agent plus token-based usage that varies by model; the page currently advertises a trial with no always-on charges (pricing).

The return depends on numbers only you have: how many alerts get investigated, how often the answer is good enough to skip manual triage (82 to 89% correct on our faults, lower on yours until you measure it), and how many engineer minutes each good answer saves. We did not time human engineers in the lab, so we have no minutes-saved figure to give you. Measure it in the pilot. Price comparisons are in AI SRE and incident tool pricing and what an AI investigation costs.

AI SRE vs AIOps vs traditional SRE: what is the difference?

AIOps products correlate and deduplicate alerts and flag anomalies; an AI SRE takes one alert and investigates it by running commands. Many teams run both. "Agentic incident response" usually names the same thing as AI SRE, sometimes with the agent also running remediation (one alert traced end to end, with the autonomy levels vendors offer). An AI SRE does not replace the SRE: in our lab both investigating agents missed a broken Service (Aurora reported the frontend workloads as unaffected), and Aurora's fix suggestions fell short in 4 of 38 runs, often enough to need a person. What it replaces is the first round of kubectl get, describe and logs. The longer comparison is in AI SRE vs AIOps.

What are the best AI SRE tools in 2026?

There is no public accuracy ranking across commercial and open-source agents, and we will not invent one. What we can say, as of 21 October 2026:

If you run on Kubernetes and want something you can try in an afternoon, HolmesGPT and Aurora are the two that did the investigation job on our faults. For a commercial product, ask the vendor to run your trial on faults you inject yourself (the lab above works for that) and grade the answers the same way. The head-to-head with features and setup is in Aurora vs HolmesGPT vs K8sGPT, the two non-Aurora tools are compared in HolmesGPT vs K8sGPT, and the open-source field beyond these three is in open source SRE agents in 2026. To try a diagnosis without installing anything, paste an error into /diagnose.

Sources and method

Lab runs: Aurora 2 October 2026 (commit 87d22baf), HolmesGPT 0.43.0 and K8sGPT 0.4.40 on 15 and 16 October 2026, same kind cluster and faults, all in Arvo-AI/aurora-lab, table in results.json. Full Aurora method and every miss: lab report v1. Vendor claims were checked against the linked pages on 21 October 2026.

Aurora is open source on GitHub; to see it on your own alerts, book a demo.