AI SRE benchmark

19 Kubernetes faults. 3 agents. Every trace public.

We break a small app on a kind cluster in known ways, hand each agent only the alert, and grade the root cause against the fault we injected. A judge model scores first, then a person checks every verdict. Misses are published with the hits. Arvo builds Aurora and wrote the scenarios; read the numbers with that in mind.

Auroracommit 87d22baf

89%

34 correct, 4 partial, 0 wrong or no answer, of 38 graded runs

median 154 s · $0.35 a run

bedrock/us.anthropic.claude-sonnet-5

HolmesGPT0.43.0

82%

31 correct, 6 partial, 1 wrong or no answer, of 38 graded runs

median 45 s

bedrock/us.anthropic.claude-sonnet-5

K8sGPT0.4.40

0 correct, 21 partial, 17 wrong or no answer, of 38 graded runs

median 13 s

us.anthropic.claude-sonnet-4-5 (K8sGPT sends temperature, which Bedrock rejects for Sonnet 5)

#	Fault	Alert the agent receives	Aurora	HolmesGPT	K8sGPT
01	Worker OOMKilled after a batch size change	KubePodCrashLooping	correct correct160 s	correct correct34 s	partial partial15 s
02	Log shipper sidecar OOMKilled, app container healthy	KubePodCrashLooping	correct correct90 s	correct correct35 s	partial partial30 s
03	Frontend rollout stuck on a mistyped image tag	KubeDeploymentRolloutStuck	correct correct89 s	correct correct25 s	partial partial17 s
04	New api pods blocked on a Secret that does not exist	KubeDeploymentRolloutStuck	correct correct164 s	correct correct30 s	partial partial14 s
05	Frontend crash loop from an invalid nginx config	KubePodCrashLooping	correct correct136 s	correct correct43 s	partial partial16 s
06	api restarts because the liveness probe targets the wrong port	KubePodCrashLooping	correct correct105 s	correct correct47 s	no_answer no_answer
07	api Service selects no pods after a selector change	HighErrorRate	correct correct157 s	correct correct47 s	partial partial13 s
08	Worker pod Pending, CPU request larger than any node	KubePodNotReady	correct correct226 s	correct correct44 s	partial partial11 s
09	api failing because redis is Pending on an unbindable PVC	KubePodCrashLooping	correct correct281 s	correct correct96 s	partial partial46 s
10	api crash loop from a wrong REDIS_HOST	KubePodCrashLooping	correct correct245 s	partial correct56 s	wrong wrong14 s
11	frontend 5xx from a wrong api Service targetPort, next to a noisy but harmless worker	HighErrorRate [critical] namespace=shop service=frontend	correct partial340 s	partial partial56 s	no_answer no_answer
12	Two faults at once, worker OOM and a frontend Service selecting no pods; the alert covers only the worker	KubePodCrashLooping [warning] namespace=shop deployment=worker	partial partial344 s	partial partial30 s	partial partial23 s
13	frontend 5xx because a NetworkPolicy blocks api to redis	HighErrorRate [critical] namespace=shop service=frontend	correct correct323 s	correct correct114 s	no_answer no_answer
14	api rollout stuck because a ResourceQuota forbids the surge pod	KubeDeploymentRolloutStuck [warning] namespace=shop deployment=api	correct correct126 s	correct correct43 s	partial partial9 s
15	One in three api requests fails because a stray Deployment's pods carry app=api	HighErrorRate [warning] namespace=shop service=frontend	correct correct147 s	correct correct130 s	no_answer no_answer1 s
16	New api pods crash because their dnsConfig search domains are for another cluster domain	KubePodCrashLooping [warning] namespace=shop deployment=api	correct correct173 s	correct correct92 s	wrong wrong16 s
17	Worker OOMKilled every few minutes by a slow leak in its code; logs look clean	KubePodCrashLooping [warning] namespace=shop deployment=worker	correct correct144 s	correct correct71 s	no_answer no_answer
18	api rollout stuck on an init container waiting for a Service that does not exist	KubeDeploymentRolloutStuck [warning] namespace=shop deployment=api	correct correct159 s	partial correct43 s	partial wrong10 s
19	Half of frontend requests fail; one frontend pod loaded an edited ConfigMap, the other still runs the old config	HighErrorRate [warning] namespace=shop service=frontend	partial correct267 s	wrong correct78 s	no_answer no_answer

How a run works

kind v0.33.0, Kubernetes v1.37.0, 3 nodes. The shop app: nginx, a Python API, a worker, redis, a load generator. Synthetic, no customer code.
One fault is injected with the scenario's break.sh; check.sh proves it is visible.
Each agent gets the alert text (K8sGPT has no notion of an alert and scans the namespace). Aurora runs under its default read-only observability_only policy; HolmesGPT and K8sGPT got the cluster admin kubeconfig, and no transcript contains a mutating command.
HolmesGPT ran through holmes ask with default toolsets and a 40-step cap it never reached; the lab has no Alertmanager, so holmes investigate was not used. Its own command allow-list blocked kubectl rollout history in 12 runs, the same problem Aurora's policy had in the first lab report.
Correct means every must_identify item in the scenario is stated and no listed wrong answer is asserted. Judge: us.anthropic.claude-opus-5-5, then a person.

Every tool ran twice per scenario. Aurora's verdicts were checked by a person. HolmesGPT’s and K8sGPT’s were graded by the judge model, then re-graded from scratch by a second model. It agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels, where K8sGPT only restated the symptom: the judge said partial or wrong, the reviewer said no answer. We kept the judge’s labels, and every grade file carries both opinions. No person has checked these 76 yet. Aurora's runs are from commit 87d22baf, before the three fixes the first lab report prompted; the six reruns on the fixed build are in the repo and not in this table.

What is not comparable: K8sGPT's row. Its analyzers flag objects and the model explains the flag, so it is scored on a criterion built for alert-driven investigators. It ran on Sonnet 4.5 because it sends a temperature parameter that Bedrock rejects for Sonnet 5; we did not try a proxy that strips it. Costs are recorded for Aurora only; HolmesGPT's token counts were not captured.

Rerun it, or add your agent

Everything is in Arvo-AI/aurora-lab: scenarios, the runner, every transcript and grade, and this table as results.json.

git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab
./lab.sh up && ./lab.sh break 03-bad-image-tag
./runner/h2h.sh 03-bad-image-tag runs/mine

To put another agent on the board, open an issue titled "Benchmark: <agent>" with a script that takes the alert text and prints the root cause. We run it on all 19 faults and publish the result, whatever it is. Submit an agent.

Read the full write-up: lab report v1. Fix the faults yourself: runbooks, each with Aurora's real trace.

Results updated 2026-10-19. Scenario and tool versions are recorded with every run in the repo.