HolmesGPT vs K8sGPT: Tested on 19 Kubernetes Faults (2026)

HolmesGPT is an open-source AI agent that investigates a production incident: you give it a question or an alert, and an LLM loop runs read-only tools (kubectl, logs, Prometheus and other data sources) until it can state a root cause. K8sGPT is a Kubernetes scanner: its analyzers list objects that look broken, and an LLM explains each finding in plain English. Both are Apache 2.0 and CNCF Sandbox projects, so the choice comes down to the job. Pick HolmesGPT to answer "why is this alert firing", and K8sGPT for a fast, cheap list of broken objects in a cluster.

We did not take that from the READMEs. In October 2026 we ran HolmesGPT 0.43.0 and K8sGPT 0.4.40 on the same 19 injected Kubernetes faults, twice each, and graded every answer against the fault we injected. HolmesGPT named the root cause in 31 of 38 runs (6 partial, 1 wrong) at a median of 45 seconds per investigation. K8sGPT's analyzers returned findings in 26 of 38 runs and printed "No problems detected" in the other 12; none of its explanations stated the full root cause, which is expected from a tool that does not investigate. These 76 grades come from a judge model, checked by a second model and by a blinded regrade. No person has reviewed them yet.

A note on bias. Arvo builds Aurora, another open-source AI SRE, and Arvo wrote the 19 scenarios. Aurora ran on the same faults; its results are on the benchmark page and in a short section near the end. Every transcript quoted here is in the public lab repo, Arvo-AI/aurora-lab, under runs/h2h-v1/.

How we tested HolmesGPT and K8sGPT

Cluster. kind v0.33.0, Kubernetes v1.37.0, 3 nodes, running a synthetic app called shop (nginx frontend, Python api, worker, redis, load generator). No customer code or data. Each fault is one break.sh in a scenario folder, and check.sh proves the fault is visible before a tool runs.

Versions. HolmesGPT 0.43.0, installed with uv tool install holmesgpt. K8sGPT 0.4.40, installed with Homebrew. Runs r1 on 2026-10-15 and r2 on 2026-10-16.

Models. HolmesGPT ran on bedrock/us.anthropic.claude-sonnet-5, the model Aurora used. K8sGPT ran on us.anthropic.claude-sonnet-4-5 through its amazonbedrockconverse backend, because K8sGPT always sends a temperature parameter and Bedrock rejects that for Sonnet 5. We did not try a proxy that strips it. That is a real difference: K8sGPT got an older model.

What each tool was asked. This is the core of the comparison, so here are the exact commands. The holmes ask and analyze lines are from runner/h2h.sh; the auth add line is the one-time backend setup from runs/h2h-smoke/README.md:

holmes ask "$ALERT Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." \
  --model "bedrock/us.anthropic.claude-sonnet-5" --max-steps 40

k8sgpt auth add -b amazonbedrockconverse -m <model> -r us-east-1
k8sgpt analyze --explain --namespace shop --output json --no-cache
k8sgpt analyze --explain --namespace shop --no-cache

HolmesGPT got the scenario's alert text (for example "KubePodCrashLooping: an api pod in namespace shop is restarting after the latest deploy."). K8sGPT has no notion of an alert, so it got the namespace. We used holmes ask with its default toolsets ("Using 28 datasources (toolsets)" in every run) because the lab has no Alertmanager for holmes investigate to read from. We did not change K8sGPT's analyzer filters. Both tools had the cluster admin kubeconfig, and no transcript shows a mutating command running.

Grading. A run is correct when it states every must_identify item in the scenario and asserts none of the listed wrong answers; partial means some items. The judge was us.anthropic.claude-opus-5-5, from the same vendor as both models under test. A second model re-read all 76 grades and agreed with all of them. A third pass regraded all 76 from scratch without seeing the judge: it agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels, calling symptom-only answers "no answer" where the judge said partial or wrong. A blinded regrade, with tool names removed, moved one HolmesGPT run (r2/19) from correct to partial and two K8sGPT runs down. We kept the judge's labels. All of it is in review2-summary.md and blind-regrade.md.

Timing. HolmesGPT's seconds are one holmes ask. K8sGPT's cover the two analyze passes above, so a single pass takes about half.

Results: every scenario, both runs

HolmesGPT's verdict is graded against the injected root cause. K8sGPT's column is the judge's label for what its analyzers and explanation said, and it measures something different: K8sGPT was scored on a criterion built for tools that investigate an alert, which it does not claim to do. Read it as "how close does the scan get", not as a failure rate. Each fault links to our runbook for it.

Costs are not in the table. HolmesGPT printed "Usage event costs for this model will be 0" because LiteLLM had no price entry for Sonnet 5, and we did not capture token counts for either tool.

Same fault, both tools: a wrong REDIS_HOST

Scenario 10 is a one-line change: kubectl -n shop set env deploy/api REDIS_HOST=redis-master. There is no Service called redis-master, so the new api pod cannot resolve the name, retries five times, exits 1 and crash-loops. The old pods keep serving. Here is run r2 for both tools.

K8sGPT printed this, in full (k8sgpt.txt):

AI Provider: amazonbedrockconverse

0: Pod shop/api-869d87b484-55cm4(Deployment/api)
- Error: Readiness probe failed: Get "http://10.244.2.129:8080/healthz": dial tcp 10.244.2.129:8080: connect: connection refused
Error: The readiness probe cannot connect to your application on port 8080. The application isn't running or listening on the expected port.

Solution:
1. Verify app runs on port 8080
2. Check container logs for startup errors
3. Ensure app starts before probe begins
4. Adjust initialDelaySeconds in probe config
1: Service shop/api()
- Error: Service has not ready endpoints, pods: [Pod/api-869d87b484-55cm4], expected 1
Error: The service cannot route traffic because its pod is not ready to accept connections.

Solution:
1. Check pod status: `kubectl describe pod api-869d87b484-55cm4`
2. View logs: `kubectl logs api-869d87b484-55cm4`
3. Verify readiness probe configuration
4. Fix app startup issues or probe settings

Everything in it is true. It found the right pod in 15 seconds and its second step ("Check container logs for startup errors") would have led a person to the answer. But the analyzer works from the readiness probe event, so the explanation is about the probe, and most of the eight suggestions (verify port 8080, adjust initialDelaySeconds, verify the readiness probe configuration) point at the probe rather than the cause. The judge and the second model called it wrong; the independent third pass called it no answer, since no cause is asserted. Either way, REDIS_HOST never appears.

HolmesGPT took 67 seconds and nine tool calls, one of them blocked. The trace, trimmed (holmes.md):

Running tool #1 kubernetes_tabular_query: filter: kubectl get pods
Running tool #2 bash: kubectl get pods -n shop -o wide | grep api; echo ---;
kubectl get rs -n shop | grep api; echo ---; kubectl get deploy api -n shop -o
yaml | head -100
Running tool #3 bash: kubectl describe pod api-869d87b484-55cm4 -n shop
Running tool #4 fetch_pod_logs: Fetch Logs (pod=api-869d87b484-55cm4,
namespace=shop)
[...]
Running tool #7 bash: kubectl get endpoints redis -n shop; echo ---; kubectl get
pods -n shop -l app=redis -o wide; echo ---; kubectl describe svc redis -n shop
[...]
  Tool 'bash' requires approval: Command requires approval. Segment(s) not in
allow list: 'kubectl rollout history deployment/api -n shop'
[...]
 • ReplicaSet api-f64486c4c (created 08:22:55, revision 1): REDIS_HOST=redis —
   2/2 pods Running, serving /healthz and /items with HTTP 200.
 • ReplicaSet api-869d87b484 (created 08:23:04, revision 2, current):
   REDIS_HOST=redis-master — 1 pod (api-869d87b484-55cm4), CrashLoopBackOff.

The only Redis Service in shop is named redis (ClusterIP 10.96.64.60, endpoint
10.244.1.145:6379, backed by pod redis-59458f55d8-vf8vz). There is no
redis-master Service.

Pod logs for api-869d87b484-55cm4 confirm the DNS failure:

 redis connection failed (attempt 1/5): [Errno -2] Name does not resolve
 ...
 giving up: redis unavailable
[...]
The latest deploy changed container env REDIS_HOST from redis → redis-master,
pointing at a Service that doesn't exist.

That is the whole difference between the two tools in one scenario. K8sGPT reports what the cluster says is broken. HolmesGPT keeps asking questions: it compared the two ReplicaSets' env, checked which Services exist, read the logs, and noticed on its own that the last-applied-configuration annotation still said redis, so the change was pushed with kubectl set env, patch or edit rather than kubectl apply (it was set env). It also hit its own command allow-list: kubectl rollout history needs approval in HolmesGPT's default bash toolset, and it routed around that by reading the ReplicaSets directly.

HolmesGPT did not get this right both times. In r1 on the same fault it saw the same two ReplicaSets and inverted them: "The latest deploy rolled the api Deployment (namespace shop) to a new ReplicaSet (api-f64486c4c) that sets REDIS_HOST=redis", called the crashing pod the "tail end of a rollout", and called the bad config "cosmetic now". The judge graded it partial. An LLM loop can read the right evidence and still draw the wrong conclusion, and you only see it when you check the trace.

Where K8sGPT's analyzers do well

When the fault is written on a single object, K8sGPT is fast and precise. On the mistyped image tag (scenario 03, r1) its first finding was:

0: Pod shop/frontend-7b6c998456-rv848(Deployment/frontend)
- Error: Back-off pulling image "nginx:1.27-alpne": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.27-alpne": failed to resolve reference "docker.io/library/nginx:1.27-alpne": docker.io/library/nginx:1.27-alpne: not found
Error: The Docker image "nginx:1.27-alpne" cannot be found because the tag name is misspelled.

That is the root cause, in 17 seconds for two passes. It was graded partial because the second finding said the frontend Service "cannot route traffic", while the old ReplicaSet was still serving. The same pattern held for the unschedulable pod (it quoted 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu, but led with "Add tolerations" rather than the 64-core request), the missing Secret (secret "api-secrets" not found) and the ResourceQuota (exceeded quota: compute-resources). On the Pending PVC it found the cause (storageclass.storage.k8s.io "fast-ssd" not found) but listed the api crash loop as a separate finding with a generic fix, never linking it to redis being down. K8sGPT points at the right object or close to it. It does not connect findings to each other or to the change that caused them.

In 12 runs it printed No problems detected: the wrong liveness port, the wrong Service targetPort, the NetworkPolicy blocking api to redis, stray pods sharing the app=api label, the slow memory leak and the ConfigMap drift, both runs each. In all six something was failing (pods being restarted in 06 and 17, users getting 5xx in the other four), but no object was in a state its analyzers flag at scan time. (networkPolicyAnalyzer is in K8sGPT's optional set, per its README, and we did not add filters.) HolmesGPT found the NetworkPolicy in both runs, the leak in both and the stray pods in both, because it started from the symptom in the alert.

Where HolmesGPT went wrong

Seven of HolmesGPT's 38 runs were not correct, and they share a shape.

• Scenario 19, r1, wrong. One frontend pod had been restarted with an edited ConfigMap (api:8000), the other still ran the old config from memory. The 502s HolmesGPT quoted came from the healthy pod's logs; it never compared the two pods, blamed "a transient startup race, now resolved", and declared the incident over. In r2 it found the drift, but said the healthy pod was fine only because the kubelet had not re-synced the ConfigMap yet (nginx reads its config only at start); the blinded regrade marked that run partial.
• Scenario 11, both runs partial. The real fault is a wrong Service targetPort, next to a worker that logs noisy but harmless errors. HolmesGPT found the targetPort and the right fix both times, but never ruled out the worker's error lines, which the scenario requires: r1 only called the worker "unaffected", r2 never mentioned it.
• Scenario 12, both runs partial. Two faults at once; the alert covers only the worker OOM. HolmesGPT diagnosed the OOM correctly and never checked Services, so it missed the frontend selector fault.
• Scenarios 10 and 18, r1 partial. The inverted rollout above. On 18 the diagnosis was right, but its first fix was to deploy the missing postgres Service, which the api does not use.

The allow-list matters here. kubectl rollout history was blocked in 12 runs, including 2 of the 7 that were not correct. We did not rerun those with the command allowed. That is a safe default and you can extend it, but out of the box it hides the one command that answers "what changed".

How do HolmesGPT and K8sGPT differ?

Checked against each project's GitHub and docs on 2026-10-21.

HolmesGPT MCP vs K8sGPT MCP

They point in opposite directions, which matters if you are wiring either into an agent you already run.

HolmesGPT is an MCP client. You add MCP servers under mcp_servers in its config, over streamable-HTTP (recommended), stdio or the deprecated SSE (remote MCP docs). Several of its built-in integrations are MCP servers: AWS, Azure, GCP, GitHub, GitLab, Jenkins, Sentry, Splunk, Prefect, Confluence, Atlassian Rovo, and "Kubernetes Remediation (MCP)", described as "Apply fixes like scaling, rollbacks, and resource edits" (README). That last one is the write path: its docs say it can restart pods, scale deployments, drain nodes and patch resources, "with every mutating action gated behind human approval". Review it before you enable it.

K8sGPT is an MCP server. k8sgpt serve --mcp runs it over stdio for Claude Desktop and similar clients, and k8sgpt serve --mcp --mcp-http serves HTTP on port 8089 by default. Its source registers 12 tools (analyze, cluster-info, config, list-resources, get-resource, list-namespaces, list-events, get-logs, list-filters, add-filters, remove-filters, list-integrations) and 3 resources (cluster-info, namespaces, active-filters). If you already use an MCP-capable assistant and want it to see your cluster through K8sGPT's analyzers, this is the cheapest way to do it.

We ran both tools as CLIs, not through MCP.

Operator mode: HolmesGPT vs K8sGPT in the background

HolmesGPT's operator mode runs health checks as custom resources: one-off, on a cron schedule, or "whenever a matching Deployment is rolled out". A "kopf-based controller handles CRD orchestration and scheduling, while stateless Holmes API servers execute the actual checks." The docs say it "is in alpha and subject to breaking changes" and warn that "Each health check triggers an LLM call (at least 1)." With the GitHub MCP server connected, "Holmes can open PRs to fix the problems it finds."

The k8sgpt-operator runs the analyzers in-cluster on an interval ("If not specified, the default interval is 30 seconds") and writes results as custom resources. It now has auto-remediation that is "deliberately alpha and opt-in": the LLM never gets write access, and the operator computes one image patch itself and dry-runs it against the API server before it applies anything.

Our test covered neither operator.

HolmesGPT vs kagent

kagent is not a HolmesGPT alternative in the way K8sGPT is. HolmesGPT is a finished SRE agent with a prompt, toolsets and a CLI. kagent is "a Kubernetes native framework for building AI agents": you define an Agent custom resource, attach tools over MCP, and its controller runs the agent and "exposes gRPC, A2A, and MCP APIs." It was created at Solo.io, which announced it on March 17, 2025 and contributed it to the CNCF; it was accepted to the CNCF Sandbox on May 22, 2025. Apache 2.0, 4,066 GitHub stars on 2026-10-21, latest release v1.0.0-alpha16 on 19 Oct 2026.

Choose kagent when you want to build and run your own agents (an SRE agent, a release agent, a cost agent) as Kubernetes resources managed with GitOps, and you are willing to write the prompts and pick the tools. Choose HolmesGPT when you want an investigation agent that works today from an alert. We did not run kagent in this benchmark, so we have no accuracy numbers for it.

When should you pick HolmesGPT, K8sGPT or both?

Pick HolmesGPT when the question is "why is this alert firing". In our runs it named the root cause on 31 of 38 attempts, read logs, compared ReplicaSets and checked Services without being told to. It is the one of the two built to reach outside Kubernetes (AWS, GCP, Azure, databases, observability tools); K8sGPT's optional AWS integration adds only an EKS check. Budget for an LLM call per step and for reading the trace: the r1 inversion on scenario 10 looked confident.

Pick K8sGPT when you want a fast, cheap inventory of broken objects: a pre-deploy check, a CI gate, a cron job that posts to Slack, or an MCP tool for an assistant. It took a median 15 seconds for two passes when it had findings, the analyzers need no LLM at all, and when a fault is written on an object (bad image tag, Pending pod, missing Secret) it names it exactly. Do not use it as an incident investigator. In 12 of 38 runs it said "No problems detected" while pods were being restarted or requests were failing.

Use both if you run Kubernetes at any size. They do not overlap much: K8sGPT tells you what is broken now, HolmesGPT tells you why an alert fired.

Where Aurora fits

Aurora is Arvo's open-source AI SRE, and it ran on the same 19 faults with the same model as HolmesGPT. A person reviewed all 38 of its grades: 34 correct, 4 partial, 0 wrong, at a median of 153.5 seconds and $0.35 per run. Its runs are from 2026-10-02, its times include contention from its own background jobs in the lab stack, and its prompt differs slightly ("Investigate and find the root cause" instead of "Find the root cause and say what changed"). HolmesGPT is about three times faster at the median and close on accuracy (31 correct, judge-graded, not yet reviewed by a person). With 38 runs each and a model judge, read the two as roughly even. The full table, prompt differences and caveats are on the benchmark page and in the lab report.

For the three-way comparison on features and deployment, see Open-Source AI SRE: Aurora vs HolmesGPT vs K8sGPT. For what HolmesGPT is and how it handles estates beyond one cluster, see HolmesGPT alternative for multi-cloud. To check a single failing pod against our runbooks without installing anything, try /diagnose.

Sources and verification

• Lab data: results.json and every transcript and grade under runs/h2h-v1. Tested HolmesGPT 0.43.0 and K8sGPT 0.4.40 on 2026-10-15 and 2026-10-16.
• HolmesGPT: GitHub, LLM providers, operator mode, MCP servers, CNCF page.
• K8sGPT: GitHub, backend registry, MCP server source, k8sgpt-operator, CNCF page.
• kagent: GitHub, kagent.dev, CNCF page.
• Project facts, stars and releases checked 2026-10-21. We retest when either tool ships a major version.

Aurora is on GitHub under Apache 2.0. To see it on your own alerts, book a demo.