HolmesGPT Review 2026: What It Is and How It Did in 38 Runs

HolmesGPT is an open-source (Apache 2.0) AI agent that investigates production incidents by letting an LLM call read-only tools against your Kubernetes cluster, metrics, logs and cloud APIs until it can name a root cause. Robusta created it, Microsoft contributes to it, and it has been a CNCF Sandbox project since October 8, 2025. It is good: when we ran HolmesGPT 0.43.0 on 19 injected Kubernetes faults, twice each, a judge model graded 31 of its 38 answers correct, 6 partial and 1 wrong, at a median of 45 seconds per investigation.

Those 38 grades come from a judge model, re-read by a second model and a blinded regrade, and no person has checked them yet. Arvo builds Aurora, a competing open-source agent, and Arvo wrote the 19 scenarios. Every scenario, every HolmesGPT transcript and every grade is public in Arvo-AI/aurora-lab, so you can check our reading against the raw output.

What is HolmesGPT?

The README calls it an "Open-source AI agent for investigating production incidents and finding root causes" that "Works with any stack". The facts a reader usually wants, checked on 2026-10-21:

"Robusta HolmesGPT" and "CNCF HolmesGPT" are the same project. Robusta still runs the commercial platform around it (the hosted UI, the Slack bot, a managed model endpoint called Robusta AI), while the agent itself lives under the CNCF.

How does HolmesGPT work?

You give it a question or an alert. The model picks tools from its enabled toolsets, reads the output, and loops until it has an answer. In our runs the CLI printed Using 28 datasources (toolsets) with default settings, then initialized only the ones the model used (kubernetes/core in 29 of 38 runs, bash in 25).

Toolsets are the core idea. The built-in list covers Kubernetes, Prometheus, Grafana, Loki, Tempo, Datadog, New Relic, Elasticsearch, ArgoCD, Helm, Kafka, RabbitMQ, SQL databases and more, with AWS, Azure, GCP, GitHub and Sentry reached through MCP servers. You can add your own as custom toolsets, REST API toolsets or remote MCP servers.

Shell access goes through the bash toolset, which checks each command against an allow list. The CLI default (core) pre-approves kubectl get, describe, logs, top, events and a few others plus text tools like jq and grep. Anything else asks you for approval, and you can widen the list in ~/.holmes/config.yaml. This matters for the results below.

The two commands you will use:

# Ask anything; HolmesGPT picks the tools
holmes ask "what pods are failing?"

# Pull active alerts from Alertmanager and investigate them
holmes investigate alertmanager --alertmanager-url http://localhost:9093

PagerDuty, OpsGenie and Jira can also be investigation sources from the CLI, with findings written back (README). Operator mode, marked alpha, runs scheduled and deployment-triggered health checks as Kubernetes CRDs and posts to Slack.

How do you install HolmesGPT?

From the CLI install docs:

# Homebrew (Mac/Linux)
brew tap robusta-dev/homebrew-holmesgpt
brew install holmesgpt

# or pipx
pipx install holmesgpt

holmes ask --help

For a cluster install, the Helm chart is robusta/holmes from https://robusta-charts.storage.googleapis.com, configured with a values.yaml that lists your models. Set the provider key (for example ANTHROPIC_API_KEY) and choose the model with the model flag shown below. HolmesGPT's own docs say "Model choice has a large impact on results" and publish per-model eval results.

Can HolmesGPT run with your own model or offline?

Yes for your own model, with a caveat for fully local ones. Any hosted provider above works with your own key, and an OpenAI-compatible endpoint (a LiteLLM proxy, vLLM and similar) covers self-hosted serving. For Ollama the docs warn that "Ollama support is experimental and can be tricky to configure correctly" and that tool calling with local models may be inconsistent. We ran it on AWS Bedrock (bedrock/us.anthropic.claude-sonnet-5), which worked without changes. One gap we hit: LiteLLM had no price entry for that model, so HolmesGPT logged usage cost as 0 and we have no cost per run for it.

How good is HolmesGPT? Our test on 19 Kubernetes faults

We built a small app (shop: nginx frontend, Python api, worker, redis) on a 3-node kind cluster (kind v0.33.0, Kubernetes v1.37.0) and broke it 19 ways. Ten scenarios have one clear fault. Nine are harder: a red herring next to the fault, two faults at once, a slow leak, a half-applied config change. Each scenario has a written ground truth and a list of findings a correct answer must contain. Full method in the lab report; live numbers on the benchmark page.

HolmesGPT 0.43.0 got the alert text and this, through holmes ask with default toolsets:

holmes ask "<alert text> Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." \
  --model "bedrock/us.anthropic.claude-sonnet-5" --max-steps 40

Aurora got the same alert text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." on the same model. The lab has no Alertmanager, so holmes investigate was not used. HolmesGPT's runs were on 2026-10-15 and 2026-10-16; Aurora's 38 runs are from 2026-10-02, on the same cluster and the same injected faults.

The results

The 3-run gap in correct answers is not statistically meaningful at this sample size: a two-sided Fisher exact test on correct versus not (34 of 38 against 31 of 38) gives p ≈ 0.52, the same test the benchmark page computes from the totals. That test also compares person-read Aurora grades with judge-only HolmesGPT grades. Read the two as roughly even on accuracy.

Speed is where HolmesGPT clearly wins. Its median is about 3.4 times faster than Aurora's, and it was faster on the median of every one of the 19 scenarios. Part of Aurora's time is contention from its own background jobs in the lab stack, but even its single-fault median (135.5 s) is three times HolmesGPT's (41 s). For a CLI you run during an incident, 45 seconds versus two and a half minutes is a real difference.

Three checks on the HolmesGPT grades. A second model (Claude Sonnet 5) read every grade next to the judge's and agreed with all 38 (review-summary.md). A third, fresh pass graded all 38 from scratch without seeing the judge's verdict and also agreed with all 38. A blinded regrade, with agent and tool names removed, matched 37 of 38 and moved one run down (scenario 19, run 2, from correct to partial). The judge (Claude Opus 5.5) is from the same vendor as the model both agents ran on. The third pass and the blind regrade are in review2-summary.md and blind-regrade.md.

What a good HolmesGPT answer looks like

Scenario 15: a stray Deployment's pods carry app=api, so the api Service sends a third of requests to a broken pod. The alert only says 31% of /api/ requests return 5xx. HolmesGPT, run 1, 97 seconds:

The shop namespace has two separate backend Deployments that both carry the Pod
label app: api:
[...]
shop-api's app.py crashes on every /items request:

 KeyError: 'DATABASE_URL'
[...]
Since kube-proxy load-balances the api Service round-robin/random across its 3
endpoints, 1 of every 3 (~33%) requests to frontend's /api/* proxy_pass hits the
broken shop-api pod and returns 500

That is a correct, evidenced answer, including the arithmetic that ties 1 in 3 endpoints to the 31% error rate. On scenario 17 (a slow leak in worker code, logs clean) it found the unbounded list in the worker's source, worked out about 5 MB per batch against the 192Mi limit, and checked that the restart changed nothing in env, image or limits. Both scenarios have runbooks built from the same lab: Service selecting the wrong pods and memory leak OOMKilled.

Where HolmesGPT missed

All seven non-correct runs, none left out:

Scenario 19, run 1, is the one wrong answer. The frontend ConfigMap's upstream was changed to api:8000, and only the pod that restarted picked it up. HolmesGPT read logs from the healthy pod, saw old 502s from startup, and concluded, in 49 seconds:

AI: All data points to a transient startup race, now resolved.
[...]
Current state (resolved): both api pods and both frontend pods are now
Running/Ready with 0 restarts, the api Endpoints object lists both pod IPs as
healthy, and current api logs show steady 200 responses on /healthz and /items.
No new 502s in the latest frontend logs.

In run 2 it found the split and the api:8000 change, though it misstated why the healthy pod was still healthy. Aurora did not do much better here: one correct, one partial. This failure pattern (one pod sampled, the answer generalized) is the subject of the ConfigMap change not applied to all pods runbook.

Scenario 12 tripped both agents in all four runs. Each diagnosed the worker OOM the alert named and stopped. If your alerts cover one symptom of a wider change, neither tool will go looking unprompted; see when the alert covers only one fault.

Was the test fair to HolmesGPT?

Things that went against it:

• Its bash allow list refused kubectl rollout history in 12 runs, 2 of its 7 non-correct runs among them. Because holmes ask ran non-interactively, it could not prompt us, so the command was denied. The docs show adding exactly that command to allow in config. We did not rerun with it allowed. Aurora's own command policy had the same block in our first lab report.
• Default toolsets only. A tuned install with Prometheus and Loki connected would see more than kubectl.
• Arvo wrote the scenarios and the must-identify lists, and builds the competing product.

Access was the same for both: each got the kind cluster's admin kubeconfig, behind its own guard (Aurora's default observability_only command policy, HolmesGPT's bash allow list). No transcript shows a mutating command running.

Things that went for it:

• Aurora's runs come from a commit dated the day before the scenarios were written, so neither tool was tuned on them.

What HolmesGPT does not do

On the evidence above it is a strong in-cluster investigator. The limits are about scope, not quality:

• Read-only by default. The README says "By design, HolmesGPT has read-only access and respects RBAC permissions." Writes need the separate Remediation MCP, where mutations go through one run_kubectl_command tool that is listed under approval_required_tools by default, so each one waits for a person.
• Its job is the investigation. The README describes fetching an alert or ticket and writing the analysis back to that source or to Slack. Incident records and postmortems stay in whatever incident tool you already run.
• Cloud reach goes through MCP. AWS, Azure and GCP are MCP-based toolsets rather than direct CLI access, so cross-cloud depth depends on those servers and the credentials you give them.
• Local models are experimental, per its own Ollama page.
• No cost figure in our runs, because LiteLLM had no price for the model. Check this on your model before you budget.

HolmesGPT vs alternatives: which should you pick?

Pick HolmesGPT if your incidents mostly live in Kubernetes and the observability stack around it, you want a fast CLI answer during an incident, CNCF governance matters to your review board, and read-only by default is what your security team will approve. Its speed and its toolset catalog are real advantages.

Pick K8sGPT if you want a scanner rather than an investigator. k8sgpt analyze with its explain option lists problems its analyzers detect across a namespace and has an LLM explain each one; it does not take an alert and chase it. In our lab it ran in a median of 12.5 seconds for two passes and mostly reported the visible symptom (a crashed container, a Service with no ready endpoints) with a checklist, which is what a scanner is built to do; it was not given the alert, so it is not scored as a peer of the two investigators. The full head-to-head is in HolmesGPT vs K8sGPT.

Pick Aurora if you need an incident platform around the investigation: alerts arriving by webhook from your monitoring tools, investigations across AWS, Azure, GCP, OVH, Scaleway and Kubernetes through their CLIs, postmortems exported to Confluence, Notion or SharePoint, and suggested fixes that can become a GitHub pull request after a human approves it. It self-hosts with Docker Compose, Helm or an air-tight bundle and takes any model, including Ollama (README). On our faults it matched HolmesGPT's accuracy and was about three times slower.

For more options, including Cleric and other self-hosted agents, see the best open-source AI SRE tools, HolmesGPT vs Cleric and the self-hosted AI SRE guide. If you only have a broken pod right now, paste its kubectl output into the diagnose tool.

Try Aurora

Run both on your own failures. Aurora is Apache 2.0 at github.com/Arvo-AI/aurora, and you can book a demo.

Tested with HolmesGPT 0.43.0 on 2026-10-15 and 2026-10-16, against Aurora runs (commit 87d22baf) from 2026-10-02. HolmesGPT facts checked against its GitHub repo, docs and the CNCF project page on 2026-10-21. Raw runs: Arvo-AI/aurora-lab.