Traditional AIOps platforms work on many alerts at once: they correlate related alerts into one incident, learn baselines and flag anomalies, and cut the number of pages a person gets. An AI SRE works on one alert: an LLM agent runs read-only commands against the live system, follows what each result shows, and returns a root cause with the evidence behind it. AIOps decides what deserves attention; an AI SRE works out why it broke. Buy for the step that is slow on your team today, and run both when you have alert floods and slow investigations at the same time.
The AI SRE complete guide has the short version of this comparison. This page shows the difference with evidence: what four AIOps products say they do in their own docs (checked 21 October 2026), one complete investigation from our lab with every command the agent ran, and the same fault seen by a scanner, K8sGPT, which lists problems instead of investigating an alert. Arvo AI builds Aurora SRE, the agent in the trace, and wrote the lab scenarios. Every run is public in Arvo-AI/aurora-lab and the totals are at /benchmark.
What does a traditional AIOps platform actually do?
Three jobs, all of them across many signals: correlation (which alerts belong to the same incident), anomaly detection (which metric is behaving unlike its baseline) and noise reduction (fewer, better pages). In each vendor's own words:
| Product | Job | What the vendor's docs say |
|---|---|---|
| BigPanda | Correlation | "Correlation patterns set rules to define relationships between system elements, which are then used to dynamically cluster alerts into incidents." Patterns use source systems, tags and a time window, and BigPanda "can correlate a maximum of 300 alerts into a single incident." |
| PagerDuty Intelligent Alert Grouping | Noise reduction | "automatically consolidates related alerts into a single, actionable incident, using machine learning to reduce alert noise". Part of the PagerDuty AIOps add-on or Signal Intelligence in PD Reliability Platform plans. |
| Datadog Watchdog | Anomaly detection | "Watchdog proactively computes a baseline of expected behavior for your systems, applications, and deployments. This baseline is then used to detect anomalous behavior." |
| Dynatrace Intelligence | Anomaly detection and topology RCA | "applies deterministic, causation-based analysis"; it "establishes baselines, detects anomalies, and identifies root cause with topology and dependency context." |
What these have in common: the input is a stream (alerts from many monitors, metrics from many hosts), and the output is a grouped incident, an anomaly, or a ranked suspect. Two of them reach toward cause over their own data: Dynatrace reasons over its dependency graph and says it "correlates code changes, deployments, configuration, and policy updates to show what changed", and Watchdog RCA draws "causal relationships between symptoms" from APM anomalies. None of the four describes running kubectl against a cluster, or reading a live object, to test a theory about one incident. That is the AI SRE's job.
The labels are converging at the product level. PagerDuty's docs list its SRE agent (Paige) next to AIOps; Dynatrace says its platform "can propose and run approved actions through agentic workflows and built-in agents" (marked Preview); Datadog sells Bits Investigation, an autonomous agent, next to Watchdog. So ask which job a feature does, not which category the vendor files it under.
What does an AI SRE agent do with one alert?
Here is a complete run from our lab, scenario 11, run 1, on 2 October 2026 (raw files). The setup is built to look like an AIOps problem. Two changes went out at about the same time:
- The worker was redeployed with code that logs a
DeprecationWarningon every batch and a metrics pushERRORon every third (break.sh). Noisy, harmless, not on the request path. - The api Service's
targetPortwas changed from 8080 to 9090. Nothing listens on 9090.
One alert fired: HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%). Aurora received that text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." It made 19 tool calls in 144 seconds, issued two or three at a time, at a model cost of $0.44:
| # | Call | What came back |
|---|---|---|
| 1 | List open incidents | The HighErrorRate incident |
| 2 | Search memory for "shop frontend" | No matches |
| 3 | List connected clusters | kind-aurora-lab |
| 4 | kubectl get pods -n shop -o wide | Every pod Running, 0 restarts |
| 5 | kubectl get events -n shop --sort-by=.lastTimestamp | Startup events, readiness failures at startup, and the worker rolled to a new ReplicaSet 58 s earlier |
| 6 | kubectl logs <frontend pod> -n shop --tail=100 | nginx startup, then connect() failed (111: Connection refused) to the api Service on 8080 and 502s |
| 7 | kubectl logs <api pod> -n shop --tail=100 | api listening on :8080, /healthz returning 200 |
| 8 | kubectl describe pod <frontend pod> -n shop | Pod detail |
| 9 | kubectl get svc,endpoints -n shop | api endpoints on port 9090 |
| 10 | kubectl logs <second api pod> -n shop --tail=100 | Same as step 7 |
| 11 | kubectl get pods -n shop -o wide | Unchanged |
| 12 | kubectl describe svc api -n shop | TargetPort: 9090/TCP |
| 13 | kubectl get deploy api -n shop -o yaml | containerPort: 8080 |
| 14 | kubectl logs <second frontend pod> -n shop --tail=30 | The same connect() failed (111: Connection refused) and 502s |
| 15 | kubectl get nodes -o wide | Three nodes Ready |
| 16 | kubectl top pods -n shop | Low CPU and memory |
| 17 | kubectl get svc api -n shop -o yaml | Last applied targetPort 8080, live 9090 |
| 18 | kubectl logs <loadgen pod> -n shop --tail=20 | GET /api/items -> 502 |
| 19 | kubectl logs <worker pod> -n shop --tail=30 | The deprecation and pushgateway noise |
The frontend log at step 6 already showed the symptom: nginx refused on its way to the api Service at port 8080. Step 9 is the first output that shows the fault. The Service port is 8080 but the endpoints are on 9090:
NAME ENDPOINTS AGE
endpoints/api 10.244.1.131:9090,10.244.2.124:9090 97s
Steps 12 and 13 confirmed it from both sides (TargetPort: 9090/TCP on the Service, containerPort: 8080 on the Deployment), and step 14 showed the same refusal on the second frontend pod:
2026/10/02 20:24:21 [error] 27#27: *49 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.2.126, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.190.97:8080/items", host: "frontend"
and step 17 showed how it happened: the last-applied-configuration annotation still says "targetPort":8080 while the live spec says targetPort: 9090, so someone edited the Service after it was applied. Only then, at step 19, did the agent read the worker's logs:
WARNING py.warnings: DeprecationWarning: StrictRedis is deprecated and will be removed in redis-py 6.0, use Redis
ERROR metrics: push to pushgateway:9091 failed: [Errno -2] Name does not resolve (will retry next batch)
Its report put them outside the causal chain, though under "Not Checked" rather than "Ruled Out", described only as pushgateway:9091 DNS errors (the DeprecationWarning is not mentioned) and "not further investigated". The report's root cause: the api Service's targetPort "was set to 9090, while the api Deployment's containers only expose and listen on port 8080". Its suggested fix was to patch targetPort back to 8080. The run was graded correct, and a person (Ben Gervais) read that grade. Runbook for this failure: Service targetPort mismatch.
Now picture the same signals reaching a correlation layer, with the worker's ERROR lines also wired to an alert (in the lab only the 5xx alert fired). The worker noise and the frontend's 5xx started together, in the same namespace, after two changes made at about the same time. Time and topology both say "related", and grouping them would be right that they arrived together and silent on which one matters. A platform that records changes, as Dynatrace says it does, would see two: a worker rollout and a Service edit. Ruling the worker out took a different kind of evidence: endpoints, ports and a config annotation. That is our reading of how the four products describe themselves, not something we tested on them. The same caution applies to us: Aurora's own alert correlation weighs topology and time, and this scenario never exercised it because only one alert fired.
What did the agent get wrong in that run?
The answer was right; parts of the report were not.
- It called the worker's errors "a separate, pre-existing cosmetic issue". They were not pre-existing: the worker had been redeployed at about the same time, and the events it read at step 5 show the new worker ReplicaSet created 58 seconds earlier. The grading caught it.
- It said frontend readiness probes were "also failing during this window". The only frontend readiness failures in the events were at startup, 66 seconds before the agent read them, and the frontend pods were Ready throughout the 502s.
- It never stated the fix in the report itself; the fix came only in its separate suggestions.
The second run of the same scenario was graded partial. It found the same port mismatch, took 535 seconds, and listed the worker under "Not Checked" with the reason "no symptoms or alerts pointed to these components, so they were not examined." Run 1 at least read the worker's logs before setting them aside; run 2 never opened them, so its dismissal was untested. It also claimed "the fix was validated but not applied" without naming a fix, and the grading marked that unsupported. HolmesGPT 0.43.0 found the port in both of its runs (65 and 46 seconds, on 15 and 16 October) and was graded partial both times for not addressing the worker noise (run 1 called the worker "unaffected", run 2 never mentioned it), on judge-model grades only.
How is a scanner different from an agent? K8sGPT on the same faults
K8sGPT 0.4.40 runs a different kind of job. Its README calls it "a tool for scanning your Kubernetes clusters, diagnosing, and triaging issues in simple English" (k8sgpt-ai/k8sgpt). k8sgpt analyze --explain --namespace shop walks Kubernetes objects through its analyzers and has an LLM explain what they flag. It never saw the alert. So we do not score it as a failed investigator; we report what its scan showed.
On scenario 11 it printed this, in both runs:
AI Provider: amazonbedrockconverse
No problems detected
Every pod was Running and Ready and every Service had endpoints, which is what its checks look at. The one wrong value, a Service targetPort that no container listens on, only shows when you compare two objects or send traffic. K8sGPT printed "No problems detected" in 12 runs across 6 scenarios (06, 11, 13, 15, 17, 19). Not all were quiet faults: in 06 and 17 pods were restarting under a firing KubePodCrashLooping alert, and the scan still came back clean.
Scenario 12 shows the scanner's strength. Two faults went out together: the worker started OOMKilling (alerted) and the frontend Service's selector stopped matching any pod (not alerted, but the frontend was down for users). Both agents followed the alert to the worker and missed the frontend in all four of their runs; Aurora's reports said the frontend "remained healthy and unaffected" and was "running normally and unaffected". K8sGPT's scan, in both runs, listed it:
2: Service shop/frontend()
- Error: Service has no endpoints, expected label app=frontend
- Error: Service has no endpoints, expected label tier=web
Its fixes there were not acceptable (it suggested labelling pods by hand), but it saw the outage the agents did not, because it looked at everything instead of following one alert. Runbook: a second fault the alert does not cover.
AIOps vs AI SRE vs a scanner: what is the difference?
| Traditional AIOps platform | AI SRE agent | Scanner (K8sGPT) | |
|---|---|---|---|
| Input | Alert and telemetry streams from many tools | One alert plus read access to tools | The cluster's current objects, no alert |
| Method | Correlation rules and learned patterns, baselines, topology | An LLM choosing each next command from the last result | Fixed analyzers, LLM explanation of what they flag |
| Output | Grouped incidents, anomalies, fewer pages | A root cause, its evidence, what was ruled out and not checked, suggested fixes | A list of objects in a bad state |
| Scenario 11 (wrong port, noisy worker) | Not tested; time and topology would link the worker noise to the 5xx | Found the port in all four agent runs; ruled out the worker in one | "No problems detected" |
| Scenario 12 (second, unalerted fault) | Not tested; it only sees the frontend outage if something alerts on it | Missed it in all four runs | Listed it in both runs |
| Typical failure | Groups things that arrived together but are unrelated, or flags a harmless anomaly | Stops at the alerted fault, adds unsupported details | Silent when objects look healthy |
| Speed and cost in our lab | Not tested | Aurora median about 154 s and $0.35 per run; HolmesGPT median 45 s | Median 12.5 s for two scan passes |
The lab totals, for context: on 19 faults, two runs each, Aurora was graded correct in 34 of 38 runs and partial in 4, every grade read by a person. HolmesGPT was correct in 31 of 38 (6 partial, 1 wrong) on judge-model grades that no person has checked; a second model agreed with all 76 HolmesGPT and K8sGPT grades, a third pass agreed with all 38 HolmesGPT grades and disputed 6 K8sGPT labels, and a blinded regrade moved one HolmesGPT run from correct to partial. HolmesGPT was about 3x faster at the median. The /benchmark page's Fisher exact test on those totals gives p = 0.52, so the accuracy gap is not significant, and it compares person-read grades with judge-only ones. Aurora's runs are from 2 October and the others from 15 and 16 October, same cluster and faults. The prompts differed: HolmesGPT got the alert plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." which names the namespace; K8sGPT got no alert. All three got the same admin kubeconfig, each behind its own guard (Aurora's read-only observability_only command policy, HolmesGPT's allow-list, K8sGPT's read-only analyze).
Our AIOps tool groups alerts but engineers still find the cause. What should we add?
An investigation layer behind the grouping. The handoff point is the grouped incident: today it lands on a person who starts with kubectl get, describe and logs; an AI SRE can start there instead and hand the person a root cause with citations to the commands it ran. In our runs, that first pass took Aurora a median of about two and a half minutes and HolmesGPT 45 seconds.
What to check in a trial, from what went wrong in ours:
- Does the report say what it ruled out and what it did not check? In scenario 11 run 2 the only sign of the gap was the "Not Checked" line. A report without that section hides it.
- Does the cited output say what the claim says? Both false details in run 1 carried citations: "pre-existing" cited the worker log (step 19) and the readiness claim cited the events (step 5), and neither output supports it. Open the cited output for every claim the decision rests on.
- Does it notice a second fault? Neither agent did. Keep a scanner or your AIOps layer watching the rest of the system, and do not let the agent's "unaffected" close the incident.
- Can it change anything? Our agents ran with read-only guards. Keep it that way until you have read a few dozen of its reports.
The AI SRE evaluation guide covers the rest, and our lab is a ready-made trial: break your own staging cluster with the same scenarios and grade any vendor's agent against the same rubric.
When is AIOps the right buy, when an AI SRE, and when both?
| Your situation | Buy first | Why |
|---|---|---|
| Hundreds of pages a night, many duplicates, on-call burning out on noise | AIOps or the grouping in your paging tool | An agent investigating every duplicate spends time and model cost on noise, and you still read every report |
Alert volume is fine, each incident takes an hour of kubectl and log reading | AI SRE | Grouping does not shorten an investigation; the trace above is the hour it replaces |
| Many monitoring tools, no single view of what is related | AIOps | Correlation across sources is its core job; an agent works one incident at a time |
| Faults that leave no bad object or metric (wrong port, NetworkPolicy, config drift) | AI SRE | They need a theory tested against config; K8sGPT printed "No problems detected" on six scenarios, these among them |
| Both of the first two | Both | AIOps decides what becomes an incident; the agent investigates each incident |
| Kubernetes, small team, tight budget | A scanner plus an open source agent | K8sGPT finds broken objects across the namespace; Aurora or HolmesGPT investigates the alert |
Our position: most teams that ask "AIOps or AI SRE?" have one of the two problems much worse than the other, and should fix that one first. Running both is right when an agent is being paged on duplicates, or when grouped incidents still wait an hour for a person.
Should we use our observability vendor's built-in agent or a vendor-neutral AI SRE?
Ask what each can reach. The built-in agents sit on the vendor's data: Datadog's Watchdog RCA, for example, requires its APM data (docs), and Dynatrace's analysis runs on its own dependency graph. That is a strength when everything is in that vendor, and a gap when the cause is in a cluster, a cloud account or a repo it does not see. A vendor-neutral agent queries the systems directly (in the trace above, kubectl against the cluster), so it needs its own credentials and guardrails. The same split answers "AI SRE vs APM": an APM tool collects traces and service metrics and shows where errors and latency are; an AI SRE reads that data, plus the cluster, cloud and repo, to work out why. Price models differ as well; the AIOps RCA pricing comparison lists what each vendor publishes. Open source agents (Aurora, HolmesGPT) and commercial ones are compared in top AI SRE tools.
Are there open source AIOps and AI SRE tools?
Yes. Keep is an open source alert management and workflow platform (MIT outside its ee/ directory); its AI correlation is marked "KEEP ENTERPRISE ONLY" on keephq.dev, and Elastic announced an agreement to acquire it in May 2025. On the investigation side, Aurora and HolmesGPT are Apache 2.0 agents, and K8sGPT is the Apache 2.0 scanner tested above. See Keep vs Aurora for the pairing and the three-way open source comparison for every lab run.
What this comparison does not show
We did not test any AIOps platform; the AIOps column comes from vendor docs, and the correlation scenario above is our reading of them. Each lab scenario fires one alert on a small kind cluster (Kubernetes v1.37.0, 3 nodes), so nothing here measures noise reduction at volume. Arvo AI builds Aurora and wrote the 19 scenarios. A person read Aurora's grades; nobody has yet read HolmesGPT's or K8sGPT's.
How Aurora fits
Aurora is the investigation layer, not an AIOps platform. It has no anomaly detection or machine-learned baselines, and it never suppresses or mutes alerts; it does link related alerts into an investigation that is already running. It investigates with read-only commands, writes the report with "Ruled Out" and "Not Checked" sections you saw above, and suggests fixes that a person approves. It runs self-hosted or hosted: github.com/Arvo-AI/aurora, or book a demo.
Sources and verification: vendor docs for BigPanda, PagerDuty, Datadog, Dynatrace, K8sGPT and Keep checked on 21 October 2026. Lab data from benchmark/results.json (generated 21 October 2026) and the run folders in Arvo-AI/aurora-lab: Aurora runs of 2 October 2026, HolmesGPT 0.43.0 and K8sGPT 0.4.40 runs of 15 and 16 October 2026. More: the lab report, agentic incident management, Service has no endpoints, /diagnose.