Best AI Incident Investigation Tools: 12 Cleric Alternatives

The alternatives to Cleric for AI incident investigation in October 2026 are Traversal, Resolve AI, Datadog Bits Investigation, incident.io Investigations, Rootly AI SRE, PagerDuty's Paige, Grafana Assistant Investigations, Dynatrace Intelligence and BigPanda among commercial products, and Aurora, HolmesGPT and K8sGPT in open source. All of them except K8sGPT start from an alert or incident and run their own queries; K8sGPT scans a cluster and explains what it finds. Only the three open source tools and Traversal (on-premises or in your own cloud account) run inside your own environment; with a local model, the open source tools need no outside service at all. Cleric publishes a price per investigated issue ($5); of the others, only Datadog and Dynatrace publish a per-investigation figure.

We tested three of these hands-on: Aurora, HolmesGPT and K8sGPT, on the same 19 injected Kubernetes faults, two runs each. The other ten are closed SaaS products that we have not run; every claim about them below was checked on the vendor's own site or docs on 2026-10-21 and is linked. Arvo builds Aurora and wrote the 19 scenarios. If you are new to the category, the AI-powered incident investigation explainer covers how these agents work; this page is the list.

Which AI incident investigation tools are there, and which investigate rather than summarize?

"Investigates" means the tool starts from an alert or incident and issues its own queries or tool calls to test a cause. "Tested" means we ran it on all 19 lab faults. Everything else was read from the vendor's site on 2026-10-21.

Two patterns stand out. The commercial products have converged on the same loop (hypothesis, query, evidence, ranked cause), so "does it investigate" no longer separates them; data reach and where the data goes do. And none of the ten commercial vendors publishes an accuracy benchmark that anyone else can rerun. Several publish customer figures, which are listed per tool below as the vendor's claims.

What did we measure on the three tools we tested?

We built a small app (shop: an nginx frontend, a Python api, a worker, redis and a load generator) on a kind v0.33.0 cluster running Kubernetes v1.37.0 with 3 nodes, and injected 19 known faults, from an OOMKilled worker to a NetworkPolicy that blocks redis to a ConfigMap edit that only one of two pods has loaded. Aurora's 38 runs were made on 2026-10-02; HolmesGPT's and K8sGPT's on 2026-10-15 and 2026-10-16, on the same cluster setup and the same faults. The full table, every trace and every miss are on /benchmark and in Arvo-AI/aurora-lab.

How to read it:

• Grading. A judge model (Claude Opus 5.5) graded every answer against the injected fault: correct means every required finding and none of the listed wrong answers. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed with every one; a third pass graded all 76 from scratch, agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels (it called them no answer where the judge said partial or wrong). No person has checked those 76 yet. A blinded regrade with tool names removed matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs.
• Aurora against HolmesGPT is roughly even on accuracy. The two-sided Fisher exact test that /benchmark computes on these totals gives p = 0.52, and it compares person-read Aurora grades with judge-only HolmesGPT grades. With 38 runs each, treat it as a tie.
• HolmesGPT is about 3x faster. Its median run took 45 seconds against Aurora's 153.5. Aurora's times include contention from its own background jobs in the lab stack, described in the lab's REPORT.md, but that does not close the gap.
• K8sGPT does a different job. It does not take an alert and investigate it; it runs analyzers over a namespace and has the LLM explain what they flag. Where the fault shows on an object, such as a mistyped image tag, it is fast and concrete. Where every object looks healthy (a NetworkPolicy block, a slow memory leak, a ConfigMap only one pod has loaded) it printed No problems detected.
• Prompts differ. Aurora got the alert plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause."; HolmesGPT got the alert plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." HolmesGPT's own allow-list blocked kubectl rollout history in 12 of its runs, including 2 of its 7 non-correct runs, and we did not rerun those. Aurora's default policy blocked the same command in 26 of its 38 runs.
• Bias. Arvo wrote the scenarios and builds Aurora. A kind cluster with one small app is not production. n = 2 per scenario does not measure variance.

Where each tested tool went wrong

Aurora's worst scenario was 12, two faults at once with an alert that covers only one. Both runs found the OOMKilled worker and its batch size change and never checked Services or endpoints. Run 1 then wrote: "Other workloads in the shop namespace (api, frontend) remained healthy and unaffected [5, 14]." Run 2 said the same in other words. The frontend Service was selecting no pods. That is a confident false statement, and the lab's model review found smaller unsupported claims (none in the root cause) in 15 of the 18 Aurora reports for scenarios 11 to 19. The runbook for a second fault the alert does not cover is built on these runs.

HolmesGPT's one wrong answer was scenario 19 (run 1), a ConfigMap edit that only the recreated frontend pod had loaded. It blamed a startup race in the api pods and closed with "All data points to a transient startup race, now resolved." It fetched logs from only one of the two frontend pods, and its log lines show the old upstream, so that was the pod still serving normally. Its second run on the same fault was correct. The runbook for a ConfigMap change not applied to all pods covers that fault.

K8sGPT's clean scans came on the faults that need an investigation rather than an object check, for example the NetworkPolicy blocking traffic scenario.

Which tools show their work and link claims to evidence?

All three tools we ran show their evidence, in different forms. Aurora's report numbers its claims and cites the tool outputs behind them (the [5, 14] in the quote above), and lists what it did not check. HolmesGPT prints each tool call as it runs (Running tool #1 kubernetes_tabular_query). K8sGPT prints each analyzer finding with its explanation.

On the commercial side, the vendors say:

• Cleric: "Every line of the fix links to the production data behind it" (cleric.ai), and "Every action logged, every investigation auditable."
• incident.io: "every finding links back to where it came from: the Slack message, the pull request, the dashboard, the log line" (trust and safety docs).
• Rootly: "Each finding comes with a confidence score and a visible reasoning chain" (rootly.com/ai-sre).
• Grafana: you "review the Assistant's hypotheses, evidence, and report" (docs).
• BigPanda's investigation swarm output includes a key evidence and correlation section (docs).

Citations make a report checkable, not correct. Aurora's scenario 12 report above cited two tool outputs, both kubectl get pods, for a claim they could not support: the Service in front of those pods was broken. In a trial, open the cited evidence for a sample of claims rather than counting citations.

What does an AI incident investigation cost per incident, including tokens?

Our Aurora figure is token cost on a small cluster with a short history. A production cluster returns longer tool outputs, so expect more tokens per run. The background discovery job cost about twice as much as the investigations in our lab ($29.70 for 30 completed discovery runs against $14.53 for the 38 graded investigations), which is the kind of line item a per-investigation price hides.

Which AI investigation tools can self-host or keep data in the EU?

• Fully self-hosted: Aurora (Docker Compose, Helm, an air-gapped bundle, local models through Ollama), HolmesGPT (CLI, Docker, Helm; Robusta also sells it "as SaaS, deploy into your own VPC, or fully self-host" per robusta.dev) and K8sGPT (CLI or operator).
• In your environment, commercial: Traversal lists "Private, on-premises deployments", "Bring Your Own Cloud" and, separately, bring your own model with self-hosted or customer-managed models (security). Dynatrace Managed runs on-premises, but Dynatrace says its agentic and generative AI "is not available for Dynatrace Managed customers" (FAQ).
• EU region, SaaS: Cleric offers US or EU hosting, and its docs say "EU hosting is not an EU-only data boundary" because some operational data is processed in the US (docs). incident.io hosts its data in GCP's Belgium region (security FAQ). Resolve AI is SaaS; its optional Satellite runs in your cluster and proxies queries (security). PagerDuty has an EU service region, though it says some data such as logs and billing may be stored outside it (docs). BigPanda runs EU processing in Frankfurt (docs).
• Cloud only: Grafana's investigations are "Not available in self-managed deployments" (docs). Rootly says plainly that it is SaaS and not the right fit if security requires fully on-premises software (llms.txt).

Cleric, Traversal or Resolve AI: how do the standalone AI SREs differ?

Sources: cleric.ai/pricing, cleric.ai/technology, Cleric security docs, traversal.com, traversal.com/security, docs.resolve.ai, resolve.ai/security. Each vendor's numbers come from its own customers and its own grading; none can be rerun from outside. Traversal has published a benchmarking method on 25 private incidents, and Cleric has written about its internal pairwise judge, which is more than most. We compared two of them in more depth in Cleric vs Resolve AI.

For "Traversal vs Datadog Bits AI", "Traversal vs BigPanda" and "Traversal vs Dynatrace": Datadog's agent investigates inside Datadog plus six listed third-party sources and is billed per credit; Dynatrace's works on data inside Dynatrace and only on its SaaS; Traversal reads many observability backends and can run in your own cloud; BigPanda is an alert correlation platform first, with an investigation swarm that can start from a major-incident template.

The tools, one by one

Aurora (tested hands-on)

• What it investigates: every incoming alert from PagerDuty, Datadog, Grafana, New Relic, OpsGenie, incident.io and others starts a background investigation that runs read-only kubectl and cloud CLI commands and writes an RCA with cited evidence, a list of what it did not check, and suggested next steps. It can open a fix PR on GitHub for a human to approve; it never merges.
• Lab result: 34 of 38 correct, 4 partial, median 153.5 s and $0.35 per run (person-read grades). Slower than HolmesGPT; missed the second fault in scenario 12 both times.
• Deploy: Docker Compose, Helm, or an air-gapped bundle; models from OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter or local Ollama (README). Latest release v1.5.10, 2026-10-16. Apache 2.0.

HolmesGPT (tested hands-on)

• What it investigates: "Open-source AI agent for investigating production incidents and finding root causes" across Kubernetes, VMs, clouds and SaaS, with about 40 toolsets and alert integrations for AlertManager, PagerDuty, OpsGenie and Jira (README). CNCF Sandbox project. Read-only by design, with an opt-in Kubernetes remediation toolset.
• Lab result: 31 of 38 correct, 6 partial, 1 wrong, median 45 s (judge-model grades, not checked by a person). The fastest investigator we tested.
• Deploy: CLI, Docker, Helm, Python SDK. Latest release 0.45.0, 2026-10-15 (we tested 0.43.0). Apache 2.0.

K8sGPT (tested hands-on)

• What it does: "a tool for scanning your Kubernetes clusters, diagnosing, and triaging issues in simple English" (README). Analyzers find problems; --explain has an LLM explain them. It has an MCP server mode, and its operator has an alpha, opt-in auto-remediation for some image failures.
• Lab result: not scored as a peer, because it takes no alert. 21 partial, 5 wrong, 12 runs with "No problems detected", median 12.5 s for two passes.
• Deploy: CLI or in-cluster operator. Latest release v0.4.41, 2026-10-21 (we tested 0.4.40). Apache 2.0.

Cleric

Investigates issues that start from "an alert, monitor, or failed deploy" and verifies changes after deploy (pricing). Integrates with Datadog, Grafana, Prometheus, New Relic, Splunk, Kubernetes, AWS, GCP, GitHub, PagerDuty, incident.io, Slack and any MCP server (integrations). Read-only by default, though its GitHub and Jira integrations can open PRs and file issues. Dedicated SaaS instance per customer; on Pro and above, a connector proxies access to private endpoints. Pricing is published: Team $700 a month, Pro $1,500, Scale $3,000, $0.50 per credit, $5 per investigated issue, 1,000 trial credits.

Traversal

"Traversal reasons causally. It builds a live model of how your systems actually behave, then investigates thousands of hypotheses in parallel to isolate root cause" (traversal.com). Reads Datadog, Grafana, Prometheus, Splunk, Dynatrace, CloudWatch, Elasticsearch, Kubernetes, GitHub, GitLab and more (integrations). The only commercial tool here that says it runs entirely in the customer's environment, with your own models. Pricing on request.

Resolve AI

"Automatically investigate alerts as they fire, gather evidence across your systems, and surface a root cause hypothesis before your on-call engineer even looks" (docs). Lists 60+ integrations across observability, AWS, Azure, GCP, Kubernetes, code hosts, Slack, Teams, PagerDuty and coding agents. Shared or dedicated cloud deployment with bring-your-own keys (security). Credit-based pricing, not published. Our Resolve AI alternative page goes deeper.

Datadog Bits Investigation

"An autonomous AI agent that investigates production issues end to end. It iteratively forms hypotheses, gathers relevant telemetry, and uses data-based reasoning" (docs). Beyond Datadog it reads GitHub, Grafana, Dynatrace, Splunk, Sentry and ServiceNow (configure). Not supported on Datadog's government sites. About 6.5 credits per investigation, from $500 per 500 credits. Datadog renamed the product from Bits AI SRE and claims it helps teams "restore services 90% faster" (product page). See the Datadog Bits AI SRE alternative.

incident.io Investigations

"It forms a hypothesis and then tests it, reading your code and querying your dashboards to confirm or rule it out" (docs). It runs on incidents rather than raw alerts: on every incident automatically if you choose, and an escalation step can wait for it before paging a human. Connects to Datadog, Grafana, Splunk, AWS, Kubernetes and code (telemetry), and to other systems through MCP connectors. Data is hosted in Belgium. No bring-your-own model today. Investigations are an add-on on Pro and Enterprise with no listed price (pricing). incident.io publishes how it measures accuracy on each customer's own incidents.

Rootly AI SRE

It "starts investigating the moment an alert fires" and "runs parallel hypothesis checks across your alerts, telemetry, recent deployments, and past incidents" (rootly.com/ai-sre). It requires Rootly Incident Response; Incident Response and On-Call are $20 per user a month each and the AI SRE is "Let's talk" (pricing). Bring your own OpenAI, Anthropic or Gemini key on Enterprise. Never auto-remediates without human sign-off.

PagerDuty Paige (SRE Agent)

Paige uses connector tools "such as querying logs, reading a file, or listing pull requests" (docs); running it as a virtual responder that starts when an incident triggers is "Currently in Early Access" (docs). Connectors cover Datadog, Dynatrace, Grafana, New Relic, Splunk, Sentry, CloudWatch, Azure Monitor, GCP, GitHub and more. PagerDuty's docs say "PagerDuty AI is included with PD Reliability Platform plans"; the pricing page lists Paige as included on Starter and above and an add-on on Professional. A fit if PagerDuty is already your incident hub.

Grafana Assistant Investigations

For "incidents that span services or need analysis across metrics, logs, traces, and profiles" (docs). Runs manually, from an investigation enrichment on an alert rule, or from IRM webhooks. Grafana Cloud only. Assistant bills on active AI users and tokens; investigations are billed only by the tokens they use, with metering that started on October 1, 2026 (pricing docs). Grafana also publishes o11y-bench, an open benchmark for LLM agents on observability tasks; it does not evaluate the Investigations product itself.

Dynatrace Intelligence

Root cause comes from a fault tree analysis over Dynatrace's dependency model: it "uses this information to quickly conduct a fault tree analysis to analyze millions of dependencies and arrive at the most probable root cause" (docs). The agentic layer runs autonomous multi-step investigations, priced at 30 AI units on average. All of it works on data inside Dynatrace, and the agentic part requires Dynatrace SaaS. See the Dynatrace Davis alternative.

BigPanda

Its core is alert correlation with change data. A "swarm of AI incident investigation agents" runs when a swarm-enabled major-incident template is used or on its own (docs). It can query MCP servers you connect (settings), and BigPanda's AI integrations cover Datadog, Dynatrace, Grafana, New Relic, Prometheus (beta) and Splunk (docs). Biggy AI is now called AI Incident Assistant. Credit plans start at 20,000 credits on one- to three-year terms, with no dollar figures published (pricing).

How do you run a fair trial of any AI incident investigation tool?

Vendor demos run on the vendor's chosen incidents. The fix is a fault you injected yourself, with the answer written down before the tool runs. The lab we used is public and works for any tool that can reach a Kubernetes cluster:

git clone https://github.com/Arvo-AI/aurora-lab
cd aurora-lab
./lab.sh up                 # 3-node kind cluster, its own kubeconfig in .kubeconfig
./lab.sh selftest           # proves every fault appears as designed
./lab.sh break 13           # resets the app, injects scenario 13, waits until visible
grep '^alert:' scenarios/13-networkpolicy-blocks-redis/scenario.yaml

Send only that alert line to the tool under trial. Then grade its answer against the must_identify and wrong_answers lists in the same scenario.yaml, which the tool must never see. ./lab.sh break <next> resets the app before the next fault; ./lab.sh down deletes the kind cluster and everything in it. The lab needs Docker with about 16 GB for the VM, kind, kubectl and Python 3. Its kubeconfig contains only the kind cluster, so it cannot touch other clusters on the machine.

Rules we followed, and would hold any vendor to:

1. Same faults, same alert text, same model where the tool allows it. Write down every difference in the prompt.
2. Run each fault at least twice. HolmesGPT was wrong and then right on the same fault; one run would have hidden either result.
3. Grade on everything the tool shows, not only the summary. Our first Aurora report undercounted fixes because the grader read only the RCA text and not the suggested next steps.
4. Include faults where the alerted object looks healthy: a NetworkPolicy, a second unrelated fault, a ConfigMap only one pod loaded. Those separate investigators from scanners.
5. Record time and cost per run, including background jobs the tool runs on connect.
6. Keep every run. Publish the misses with the hits.

A SaaS product running in the vendor's cloud may not reach a kind cluster on a laptop. In that case, deploy the lab app (app/shop.yaml) on a disposable cluster the vendor's trial can reach and run the same break.sh scripts there; each one is a short kubectl change against your current context, so check kubectl config current-context first and read the script before running it. Give the tool under trial the access you would give it in production, not the lab's admin kubeconfig, unless the cluster is disposable. To add a tool to the public table, open an issue with the "Benchmark: add an agent" template in the lab repo. For a single broken pod, the Kubernetes pod diagnoser takes pasted kubectl output and returns the most likely cause and the command that confirms it, in your browser.

Which tool should you pick?

• You want an open, self-hosted investigator and can operate it: Aurora or HolmesGPT. They were roughly even on accuracy in our lab; HolmesGPT was about 3x faster, Aurora triggers on alerts from your existing tools and drafts fix PRs for approval. The open source SRE agents comparison covers six more projects.
• Data cannot leave your network but you want a vendor: Traversal is the only commercial investigator here that says it runs entirely in your environment. Dynatrace Managed runs on-premises, but Dynatrace's agentic AI does not run on it.
• Your telemetry is already in one platform: that platform's agent (Datadog, Dynatrace, Grafana) sees the most with the least setup, and sees less outside it.
• You want an incident platform with investigation built in: incident.io, Rootly or PagerDuty.
• You want a published price per investigation: Cleric ($5), Dynatrace (about $3 average), Datadog (about 6.5 credits, roughly $6.50 at the annual rate).

Whatever you shortlist, run the trial above before you sign. Aurora is on GitHub under Apache 2.0, or book a demo to see it on your own alerts.

Vendor facts verified on each vendor's site or docs on 2026-10-21. Lab numbers from results.json, generated 2026-10-21. Methodology and every trace: /benchmark and the lab report.