A benchmark result is going around that sounds like a reason to buy something.
Give an AI agent a tidy read-only view of your Kubernetes cluster instead of a terminal, and it diagnoses faults about four times faster. The number is real.
But look at the column nobody quoted and the agent got almost exactly the same number of answers right.
That gap matters to anyone being shown a demo this quarter.
By the end of this you will know what the speedup actually buys you, why it happens, and how to measure your own agent in an afternoon for free.
A word on where it comes from, since it shapes how much weight to give it. The study was published by Radar, which sells the tool that wins, and later ran on the CNCF blog as a Member Post, which means contributed by a company rather than research by the foundation. Credit where due, though. The author published his caveats, the transcripts, and the runs that went against him, which is the only reason the interesting part is visible at all.
So let's take him at his word and read the whole table. Correct diagnoses went from 45 out of 50 to 46 out of 50. One fault. His own methodology notes that on a single run per fault, a gap of two or three can be random, which marks his headline accuracy result as noise by his own standard.
What does four times faster actually buy you?
Look closely at what was timed, which was how long the agent took to submit a diagnosis, whether or not it was any good. Being wrong doesn't disqualify a run. It just means a confident answer arrived sooner.
You can see it in which faults both versions failed. Both missed a webhook silently rewriting pod memory limits as they were created. And on a DNS problem, both found a genuine but unrelated misconfiguration and reported it with total conviction. Same model, same blind spots, arriving four times sooner.
Which changes what "too slow" means. The old complaint was that investigations take so long nobody waits for them. The new problem is the wrong answer landing before anyone has finished reading the alert.
That matters more than it sounds, since a fast wrong answer sets the direction for everyone who joins the call afterwards. The SREGym paper that these faults come from pins the habit down: agents "always treat the first plausible anomaly as the target failure". In several runs the agent found evidence of the real cause and set it aside because it didn't fit the theory it had already committed to.
Nobody's dashboard measures time-to-wrong-answer. Every vendor page measures time-to-answer.
Why the context version is so much faster
The mechanism is the strongest part of the study. On one fault, a demo shop application lost the setting that tells its web page where to find the shopping cart service.
The agent with only a terminal went the long way round, pulling the application's compressed deployment record out of the cluster, decoding it, unzipping it, and comparing it by hand to see what had changed. Almost sixty commands and the best part of twelve minutes. The agent with pre-assembled context got there in seven commands and under a minute.
The same shape repeated across the whole run. Given context, the agent asked "what is broken right now?" first, every time, and "what changed recently?" second on most faults. Given a terminal, it listed pods, then went spelunking through deployment history trying to reconstruct a timeline.
And that timeline often no longer exists. Kubernetes keeps event history for about an hour by default, a setting called --event-ttl, and many configuration changes never generate an event at all. It is a shop security camera that overwrites itself hourly, and the agent was trying to reconstruct yesterday from footage already taped over. Pre-assembled context keeps a written logbook instead, and closing that gap is a genuine win.
Though it is fair to ask how much of this is "context" as a principle and how much is one very specific shortcut, since the winning product includes a tool that root-causes a single workload in one call, and it appears in the transcript above.
Your choice of tool only matters on the quiet faults
Radar ran a second benchmark comparing six tools on the same faults, and almost nobody mentioned it. The league table is the least useful thing in it. The split underneath is the part that matters.
Divide the faults into two piles. In the first, something is visibly on fire: a container restarting in a loop, an image that won't download. All six tools handled those comfortably, within a few points of each other. In the second pile sit the 14 faults where every component reported itself healthy and the application was broken anyway, and there the same six ranged from solving most to solving barely a third.
So before comparing vendors, work out which pile your incidents fall into. If your outages announce themselves, almost any of these will do. If your worst nights are the ones where every dashboard is green and customers are complaining, that second pile is the only comparison worth running.
One detail from that table deserved more attention than it got. The agent offered by the largest cloud provider refused one in ten faults outright, blocked by its own safety filter before it could begin.
Why 46 out of 50 should make you suspicious
Set that score next to everyone else's and it looks odd. On ITBench-AA, an independent evaluation built by Artificial Analysis with IBM Research, no frontier model got even half the incident-response tasks right. Yet on the public SREGym leaderboard, third parties score full marks against a small hand-picked subset of the same faults.
Which is the real lesson... which faults you run matters more than which tool you run them with. This benchmark doesn't say how its faults were chosen, and every published name I could trace lands inside the curated short list whose own documentation calls it easy and reliable to run. Suggestive rather than damning, but a good reason to ask any vendor which faults their demo used.
What the terminal got up to while nobody was grading it
The most interesting material in the study has nothing to do with speed. It is in the transcripts of the arm that lost.
Handed a terminal and told to fix things, the agent did not confine itself to looking. It started its own pods, opened shells into running containers, forwarded network ports, and read secrets into its working memory. On one fault it launched a pod whose job was to delete two of the application's live database collections, purely to test a theory, then submitted the wrong diagnosis anyway.
Nothing malfunctioned, which is the part worth sitting with. The instructions told the agent a repair stage was coming, that it could modify things, and not to stop and ask. It did as it was told. An agent with write access and a mandate to fix will use that access while it is still guessing, and guessing is most of what diagnosis is.
Which brings us to the uncomfortable part. For many teams "read-only" is something someone said during setup, not a boundary anyone has tested. Kubernetes ships a built-in view role that really withholds secrets and shell access, but hand-written versions routinely hand both back without anyone noticing.
The software connecting agents to clusters has already got this wrong twice in public. In CVE-2026-46519 the permission check ran when the agent asked what tools existed but not when it used one, so an agent restricted to reading could still call delete. Hiding a tool from the menu is not the same as removing it from the kitchen.
The second, issue #1415, is almost funny. An agent in read-only mode asked an innocent question about its own configuration, got back a file containing full credentials, and logged in with those instead. Not an agent going rogue, just an agent taking the shortest route to what it was asked to do.
The afternoon's work that should come before the purchase order
The most useful part is this. The evaluation method is sound, and worth running even if you buy nothing, since it tells you what the agent you already have can do.
- Give every contender the same model and identity. Otherwise you are comparing models, not tools. Bind a service account to that built-in
viewrole and delete the admin entry from the credentials file, so nothing can fall back to full access unnoticed. - Prove the read-only claim instead of trusting it. Run
kubectl auth can-i get secretsand insist on the answerno. That is the difference between a boundary and a sentence in a config file. - Run the public faults locally first. The short SREGym list runs on kind, a throwaway cluster on one machine with 8 vCPU and 16 GB of memory. No cloud bill, no staging to borrow.
- Run every fault more than once, and let a model none of your candidates use do the grading. The accuracy claim here rests on a single run each, which is how you end up certain of a difference that was really just luck. And if your judge is also a contestant, you have built a conflict of interest into your own evaluation.
- Record the spread, not just the average. This study shows why: its average diagnosis took a touch over three minutes, but the slowest tenth took ten minutes or more. The average described almost nobody's actual night.
The command to start it runs the short fault list against one agent and one model. The suite also takes flags for your judge model, repeat attempts, and resuming an interrupted run:
uv run main.py --suite sregym-lite --agent claudecode --model claude-sonnet-5Then do the part no benchmark can do for you. Replay two or three of your own past incidents, ones whose true cause you already know, with that same read-only identity. Benchmark faults get injected minutes after a fresh deployment, which flatters any tool that watches for recent changes. Your three-week-old memory leak will not look like that.
And change what you write down. Count how many of the agent's findings arrived with evidence you could check, against how many were confident prose. That ratio tells you something. The clock doesn't.
One last figure, since it reframes everything above. Catchpoint's SRE Report 2026 asked several hundred practitioners whether AI had reduced their grunt work.
Just under half said yes, and the rest split evenly between nothing changed and it got worse. The managers were also markedly more optimistic than the engineers carrying the pager, which is its own small lesson about who to ask.
That is the asymmetry to hold onto. A 44-second diagnosis is only worth having if someone can check it in 44 more. So this week, set up the read-only identity, run the short fault list on a laptop-sized cluster, and have a disinterested model grade the results. You will know what the agent you already own can do, which is a far better place to negotiate from than anyone's benchmark.