Evaluate
KLRB
A Kubernetes benchmark that measures whether an LLM actually read the cluster evidence before diagnosing an incident — or just guessed confidently from metadata.
Try the ablation
One Kubernetes incident, three conditions. Take away the evidence that justifies the model's answer and watch what the answer does. This takes about ten seconds and it is the entire finding.
A worked example of the ablation the harness runs. The three conditions and the evidence shape are the real ones; the response text is representative of the OOMKill scenario rather than a verbatim published run.
Why this measurement and not accuracy
You can score an incident responder on whether its answer matched the label. That
number looks fine and tells you almost nothing, because on a corpus of common
Kubernetes failures a model can reach it from priors alone —
CrashLoopBackOff on a pod with a low memory limit is an OOMKill
often enough that guessing pays.
An accuracy score cannot separate a responder that read your cluster from one that recognised the shape of your pod name. In production that difference is everything: the first one generalises to your weird outage, the second one is confidently wrong precisely when you need it most.
So KLRB scores the dependency instead of the answer. Run the scenario with the causal evidence, run it without, and compare. If the conclusion survives the removal of its own justification, the justification was decorative. I call the failure mode the Confident Liar effect — not because the model intends to mislead, but because fluency and groundedness turn out to be independent, and only one of them is visible in the output.
How it is built
- Injects reproducible chaos faults into live Kubernetes clusters, then scores how faithfully an LLM agent diagnoses root cause from the real evidence: events, logs and metrics.
- The measurement is an ablation, not a vibe check. Run the same scenario with the causal evidence present, then with it removed, and compare the conclusions.
- That ablation surfaces what I call the Confident Liar effect: a model that keeps its conclusion unchanged after the evidence that justified it is taken away was never reading the evidence. It was guessing, fluently.
- A multi-provider runner behind one adapter interface, with deterministic scenario replay — so cross-model scores are comparable instead of run-to-run noise.
- This is the most-starred thing I have built, and the reason is the metric rather than the code: "did it read the logs" turns out to be measurable.
Scope — what this is not
A benchmark harness and a reproducible effect, not a peer-reviewed result. Scores are comparable within the harness. I would not yet quote an absolute number as a property of a model in general — the honest claim is the direction and the method.
The open questions I would want a reviewer to press on: scenario count and seeds per condition, whether the ablation control removes the evidence or merely corrupts it (both are implemented, they measure different things), and how much of the effect is the model versus my prompt's log formatting. A preprint is in progress and these are the sections that matter.
Elsewhere in the system
Replaces the shared API key an AI agent uses with a scoped, attenuable token bound to its workload identity — verified offline, with no network hop on the authorization path.
An agentic incident-response pipeline: it detects a live service failure, investigates it with a three-tool orchestrator, and proposes a root cause — but cannot touch anything until a human approves.
Terraform tells you what drifted. This tells you who changed it, when, and from where — by correlating the plan against CloudTrail.