HolmesGPT Alternatives, Tested: Aurora vs K8sGPT on 19 Faults

The two open source alternatives to HolmesGPT worth testing side by side are Aurora and K8sGPT, and they do different jobs. On the same 19 injected Kubernetes faults, two runs each, Aurora named the root cause in 34 of 38 runs and HolmesGPT 0.43.0 in 31 of 38, a gap that is not statistically significant (two-sided Fisher exact test on those totals, p = 0.52, the same test /benchmark computes). HolmesGPT was about 3x faster at the median, 45 seconds against Aurora's 154. K8sGPT 0.4.40 does not investigate an alert at all: it scans a namespace and explains what its analyzers flag, in a median of 13 seconds for two analyze passes, and it printed "No problems detected" on 6 of the 19 faults (12 of its 38 runs).

Read those numbers with three facts attached. Arvo builds Aurora, and Arvo wrote the scenarios and the grading criteria. Aurora's 38 verdicts were checked by a person at Arvo. The 76 HolmesGPT and K8sGPT verdicts are judge-model grades, re-read by two other models and regraded blind, and no person has checked them yet, so the p-value above compares person-checked Aurora grades with judge-only HolmesGPT grades. Everything is public: the live benchmark table, and every scenario, transcript and grade file in Arvo-AI/aurora-lab.

Aurora vs HolmesGPT vs K8sGPT: results on 19 Kubernetes faults

"Correct" means the answer stated every finding the scenario requires and asserted none of its listed wrong answers. "Partial" means some of the required findings. K8sGPT's row uses the same rubric, but the rubric was built for tools that investigate an alert, and K8sGPT is not one. Its partials are mostly "it flagged the broken object and explained the flag, without saying what changed", which is what it is designed to do.

Our position after running all three: for alert-driven root cause analysis on Kubernetes, Aurora and HolmesGPT are roughly even on accuracy in this lab, HolmesGPT is clearly faster and lighter to run, and the reasons to pick one over the other are scope, deployment model and workflow, not a 3-run difference in correct answers. K8sGPT is a scanner that belongs next to either of them, not instead of them.

Results for every scenario

Each fault name links to the runbook we wrote from the lab trace for that failure. Aurora and HolmesGPT columns show correct runs out of 2 and the median time. Per-run verdicts and links to the raw output are on /benchmark.

Scenarios 01 to 10 have one clear fault each. Scenarios 11 to 19 were written to be harder: noise next to the fault, two faults at once, a slow leak, a partial restart. All the non-correct runs for both investigators are in the harder set except HolmesGPT's run on 10.

Where did Aurora miss?

Aurora was not correct in 4 runs, all partial.

Scenario 12, both runs: it found the alerted fault and scoped the incident to it without checking. The alert named only the OOMKilled worker. A second fault, a frontend Service whose selector matched no pods, had taken the frontend down for users. Both Aurora runs diagnosed the worker correctly (BATCH_SIZE raised from 500 to 2500 against a 192Mi limit, with the buffer math from worker.py) and neither ran get svc or get endpoints. Run 1 then wrote this, citing only two get pods calls:

Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14].

Run 2 wrote "Impact is scoped to the single-replica worker deployment in shop". That is the worst kind of miss for an incident tool: a real outage that the report says is not there. HolmesGPT made the same miss on both runs of this scenario. The runbook for a second fault the alert did not cover shows the traffic-level checks that would have caught it.

Scenario 19, run 1: the right clue, the wrong conclusion. Aurora found the edited frontend-nginx ConfigMap pointing at api:8000 while the Service listens on 8080. It did not work out that only the recreated pod had loaded the edit. It tried to read the live config inside the failing pod, and its own default command policy refused:

$ kubectl exec frontend-76d98597b9-g5zrg -n shop -c frontend -- cat /etc/nginx/conf.d/default.conf
Command blocked by organization policy: Mutating kubectl operations

So it left the question open in the report:

This discrepancy was not fully reconciled — the live nginx configuration inside the running pod could not be inspected to confirm which port it was actually using at the time of each failure, since direct exec access was blocked by policy [10].

Its own get pods output already held the answer: the failing pod was 9 seconds younger than the healthy one. Run 2 got it right with 28 tool calls. See the runbook for a ConfigMap change not applied to all pods.

Scenario 11, run 2: graded partial for not ruling out the red herring. Aurora found the Service targetPort of 9090 against a container listening on 8080, which is the fault. The scenario also has a worker logging harmless errors, and the rubric requires the answer to say those are not the cause. Aurora's report said only this about the worker:

- Health and configuration of the redis, worker, and loadgen deployments — no symptoms or alerts pointed to these components, so they were not examined.

You can argue that is a correct answer that was not misled. We kept the rubric as written, and the same rule cost HolmesGPT both of its runs on 11.

Aurora's time column is also worse than it should be. During these runs, Aurora's own background "prediscovery" jobs competed with the investigations for worker slots, which pushed some runs to 444 to 548 seconds. The lab report has the details and the fixes we made after it.

Where was HolmesGPT better than Aurora?

Speed, everywhere. HolmesGPT was faster on every one of the 19 scenarios, with a median of 45 seconds against 154. Part of Aurora's gap is the lab contention above, but not all of it: on the ten single-fault scenarios Aurora's medians are 89 to 281 seconds against HolmesGPT's 25 to 96, and no Aurora run in that set finished under 60 seconds.

Same evidence in a fraction of the time. On scenario 12, run 2, HolmesGPT read the worker's code out of its ConfigMap and did the memory math in 34 seconds. Both Aurora runs did the same math, in 140 and 548 seconds, and both tools missed the second fault:

Why it crashes: worker.py (from ConfigMap api-code) allocates batch buffers of
100 KiB each on every loop iteration: buf = [bytearray(100*1024) for _ in
range(batch)]. At BATCH_SIZE=500 that's ~50MB — fine. At 2500 that's ~250MB,
which blows past the 192Mi limit immediately.

Evidence quality on scenario 19. On run 2 it worked out the two-pod split from creation timestamps and logs, which is exactly what Aurora's run 1 failed to do:

With 1 of 2 replicas permanently broken and the other still fine, this produces
almost exactly the observed ~48% 5xx rate on /api/ (one healthy, one failing,
roughly 50/50 traffic split).

The same answer then got the mechanism wrong, and the blind regrade moved it from correct to partial. nginx reads its config only at start, so the healthy pod would not break on its own:

xmkdj is only "healthy" because it hasn't yet re-synced its mounted ConfigMap
volume (kubelet syncs mounted ConfigMaps periodically, default ~1 minute).

Setup. HolmesGPT is a CLI you install with Homebrew or pipx and point at a kubeconfig and an LLM key. Aurora is a multi-container stack (see setup below).

Where did HolmesGPT miss?

Seven HolmesGPT runs were not correct. Two of them are the scenario 12 miss described above, and two are the scenario 11 red-herring rule.

Scenario 19, run 1: wrong, and it declared the incident over. With 48% of requests failing, HolmesGPT fetched logs from one of the two frontend pods (the healthy one, judging by the upstream port in its errors), found two 502s from startup, and concluded:

AI: All data points to a transient startup race, now resolved.

It listed both frontend pods but never read the other pod's logs or the ConfigMap's contents.

Scenario 10, run 1: right symptom, inverted change. It found redis-master failing to resolve, then decided the crashing pod was a leftover from the old ReplicaSet and the latest deploy was the fix: "this is cosmetic now since the new RS already has the correct value". The judge marked the second required finding partial for that reversal.

Scenario 18, run 1: correct diagnosis, flagged fix. It named the wait-for-db init container and the missing postgres Service, which is the fault. Its first suggested fix was to deploy a postgres Service or StatefulSet. The scenario lists "creating a postgres database as the fix without noting the api does not need one" as a wrong answer, because the app does not use postgres. That made the run partial. A reader could call this strict, as with Aurora's run on 11.

HolmesGPT's own command allow-list blocked kubectl rollout history in 12 runs, 2 of its 7 non-correct runs among them. We did not rerun those with the command allowed. Aurora's default policy had the same block in the first lab report (26 of 38 runs), so neither tool had an easy "what changed" view on the default settings.

What did K8sGPT surface?

K8sGPT runs a fixed set of analyzers over the cluster and asks the LLM to explain each finding. It got no alert. That makes it fast and literal, and it is very good when the broken object carries its own error. On scenario 3 it named the typo in 17 seconds:

- Error: Back-off pulling image "nginx:1.27-alpne": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.27-alpne": failed to resolve reference "docker.io/library/nginx:1.27-alpne": docker.io/library/nginx:1.27-alpne: not found
Error: The Docker image "nginx:1.27-alpne" cannot be found because the tag name is misspelled.

On scenario 12 it was the only tool to report the second fault, in both runs, because it scans everything instead of following the alert:

2: Service shop/frontend()
- Error: Service has no endpoints, expected label app=frontend
- Error: Service has no endpoints, expected label tier=web
Error: The Service cannot find any Pods because none have the required labels (app=frontend AND tier=web).

Both investigators would have been correct on 12 if they had done what K8sGPT does by default. That is the strongest argument in this data for running a scanner next to an agent.

Where nothing looks broken at the object level, it finds nothing. On scenarios 6, 11, 13, 15, 17 and 19 it printed this in both runs, while the alert was firing:

No problems detected

Where it did report something, it reports the symptom and a generic checklist, and it does not say what changed. On scenario 1 it found the OOMKill and suggested raising the limit (to an example "2Gi" in run 1), without the BATCH_SIZE change. On 16 it blamed probe timing for what was a DNS search-domain fault, which the judge graded wrong.

Two caveats specific to K8sGPT. We ran it with its default analyzers. Its README lists networkPolicyAnalyzer and logAnalyzer as optional, and they were off; we did not test whether they change the result on 13 or 17. And the from-scratch regrade (the third pass described in the method section; the first reviewer model agreed with all 76 grades) disputed 6 of the judge's K8sGPT labels, saying symptom-only answers on 05, 10 and 18 should be "no answer" rather than partial or wrong. We kept the judge's labels, which are the more generous reading for K8sGPT, and published the dispute in the lab repo.

How fast and how expensive is each open source SRE agent?

We do not have a per-run cost for HolmesGPT or K8sGPT, so we do not publish one. HolmesGPT printed the reason at the top of every run: its LLM library had no price entry for the model.

Model 'bedrock/us.anthropic.claude-sonnet-5' has no entry in litellm's cost map
and no input/output_cost_per_token configured. Usage event costs for this model
will be 0.

Aurora has one cost that does not show up in the per-run number. Each kubeconfig upload schedules a background prediscovery run, which cost about $1 per cluster connection in the lab, almost twice what all 44 investigations together cost ($30.38 against $16.77). That is in the lab repo's REPORT.md and is something we are fixing.

How hard is each one to set up?

This is how the lab ran HolmesGPT and K8sGPT, from runner/h2h.sh in the lab repo. Each is one command against a kubeconfig:

k8sgpt analyze --explain --namespace shop --output json --no-cache
k8sgpt analyze --explain --namespace shop --no-cache
holmes ask "$ALERT Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." \
  --model "bedrock/us.anthropic.claude-sonnet-5" --max-steps 40

• K8sGPT is a single Go binary (Homebrew, RPM, DEB, APK or Windows builds), configured with k8sgpt auth add. An operator runs it in-cluster on a schedule.
• HolmesGPT installs with Homebrew, pipx or Docker (install docs), and also ships a Helm chart, an HTTP server and a Python SDK. The CLI needs only an LLM key and cluster access.
• Aurora is a self-hosted application with a web UI, a database, Vault and background workers. The quick start is make init, an LLM key in .env and make prod-prebuilt, then a Vault token step and a restart. Production installs use the Helm chart. Alerts arrive by webhook and each one starts an investigation in the background, so it is built to sit in the incident path rather than be run by hand.

If you want an answer in a terminal in five minutes, HolmesGPT and K8sGPT win. Aurora's setup pays off only if you want the alert-to-postmortem workflow it is built around.

Which open source SRE agent works beyond Kubernetes?

Our lab tests only Kubernetes. Scope beyond it comes from each project's own README and docs, checked on 2026-10-21.

Two points to weigh. HolmesGPT is not Kubernetes-only: its README says it works without Kubernetes, against VMs, cloud services and databases, and its list of observability and data toolsets is long. Aurora's difference is that it is an incident system rather than a tool: alerts come in by webhook, it investigates in the background, keeps an incident history, writes the postmortem and proposes the fix PR. Aurora's defaults run a single agent; a multi-agent orchestrator exists but is off unless you enable it.

On permissions: all three tools got the same kind admin kubeconfig (kubernetes-admin). Aurora ran under its default read-only observability_only command policy on top of it; HolmesGPT's own command allow-list asked for approval on commands it did not know, including every kubectl exec it tried, and no approval was given; K8sGPT's analyze only reads. No transcript shows any of them running a mutating command. We did not test any tool on a restricted kubeconfig. HolmesGPT's README says it has read-only access and respects RBAC, and Aurora adds its own command policy on top of RBAC.

What other open source SRE agents are there?

We have not benchmarked these. Facts are from each repository on 2026-10-21.

kagent and OpenSRE are frameworks: you build the agent. Aurora, HolmesGPT and K8sGPT ship one. If you maintain any of these and want them on the board, the benchmark takes a script that reads a kubeconfig and an alert and prints a root cause; we run it on all 19 faults and publish the result.

How was the benchmark run, and what are its limits?

• Cluster. kind v0.33.0, Kubernetes v1.37.0, 3 nodes. A synthetic shop app: nginx frontend, Python api, worker, redis, load generator. No customer code or data. Each fault is one break.sh in the scenarios folder, and check.sh proves it is visible before any tool runs.
• Dates. Aurora ran on 2026-10-02. HolmesGPT and K8sGPT ran on 2026-10-15 and 2026-10-16. Aurora's commit predates the scenarios, so it was not tuned on them.
• Prompts differ. Aurora got the alert plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." HolmesGPT got the alert plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." HolmesGPT's prompt names the namespace and asks what changed, which if anything helps it. K8sGPT got no alert.
• Models. Aurora and HolmesGPT both ran Claude Sonnet 5 on Bedrock. K8sGPT ran Claude Sonnet 4.5 because it always sends a temperature parameter, which Bedrock rejects for Sonnet 5; we did not try a proxy that strips it.
• HolmesGPT settings. holmes ask with default toolsets and a 40-step cap it never reached. The lab has no Alertmanager, so holmes investigate was not used.
• Grading. A judge model (Claude Opus 5.5) marked each required finding as met, partial or missed and quoted the answer. The judge is from the same vendor as every model under test. A second model re-read all 76 HolmesGPT and K8sGPT grades and agreed with all of them. A third pass graded those 76 from scratch without seeing the judge: it agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A person at Arvo reviewed Aurora's 38 and changed none.
• Blinded regrade. To test whether the judge favoured Aurora, every answer was graded again with the agent's name, its tool names and the cluster name removed. The blind verdict matched for 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. All three changes were downgrades of non-Aurora runs. The blind judge still recognised K8sGPT's output format, so blinding is incomplete for that tool.
• Reruns and one correction. Scenario 19 was rerun after its break script hit a TLS timeout against the API server, and scenario 18 after HolmesGPT hit a Bedrock timeout; both are logged in the repo. The first K8sGPT grade for scenario 18, run 1 was of the replaced first-pass output. We regraded the actual file on 2026-10-21 and the verdict stayed partial.
• What it cannot tell you. Two runs per scenario do not measure variance. One small app on kind is quieter than production and has no dashboards, runbooks or history, which every tool here could use. The scenarios were chosen by the vendor of one of the tools. HolmesGPT 0.44.0 (2026-10-14) and 0.45.0 (2026-10-15) were already out when our HolmesGPT runs started, and K8sGPT 0.4.41 shipped on 2026-10-21; we tested 0.43.0 and 0.4.40 and have not retested.

Which should you pick?

• Kubernetes-centric team, wants answers fast, happy to run it from a terminal or Slack: HolmesGPT. In our lab it matched Aurora within noise (its grades are judge-model only) and was about 3x faster.
• Wants a periodic cluster scan that catches broken objects nobody alerted on: K8sGPT, next to whichever investigator you choose. It was the only tool that saw the second fault in scenario 12.
• Wants alerts from PagerDuty, Datadog or Grafana investigated automatically, across Kubernetes and cloud accounts, with postmortems and fix PRs behind approval, all self-hosted: Aurora. Expect a heavier install and slower investigations than HolmesGPT.

All three are open source, so the cheapest way to decide is to break something on a test cluster and run them yourself. The lab repo breaks a scenario and runs HolmesGPT and K8sGPT against it in three commands (Aurora needs its own install first), and the Kubernetes pod diagnoser handles the one-off case where you just want to paste kubectl output.

Aurora is on GitHub at github.com/Arvo-AI/aurora; to see it on your own alerts, book a demo.

Benchmark data from results.json, generated 2026-10-21. Competitor features, versions and licenses verified against their repositories and docs on 2026-10-21.