Ask a team running an agent in production whether last week’s change made it better. Most can’t say. They swapped a model or rewrote a skill, eyeballed a few runs, and moved on.
That works for a demo. It doesn’t work for an agent that touches production, where customers want two things that pull against each other: precision where a mistake is expensive, and low cost where the work is routine. A root cause someone acts on has to be right. A check that runs every few minutes has to be cheap. And the cheap version can never quietly get worse.
You can’t guess your way to that trade-off; you have to measure it. On the Komodor Agentic Operations Platform, we do it with what we call the Evaluation Factory: a loop that finds, builds, and proves better versions of an agent, while a person decides what ships. Tracing, LLM judges, golden scenarios, and shadow runs are live today. The fully automated loop is what we’re building now.
Why are agents so hard to test?
A function returns the same value for the same input. A model can phrase its answer differently on every run. An agent can take a different path on every run, and change real systems along the way.
| Traditional software | A model | An agent | |
| What you check | A return value | An answer | Dozens of steps, tool calls and side effects |
| How you grade it | An assertion | A rubric or LLM judge | Code, judges and people, on outcome and path |
| How it fails | Loudly | A wrong answer that sounds right | An early mistake compounds, and it still reports success |
That last cell is what hurts in operations. An agent that misreads a log at step 3 can call ten more tools, write a confident root cause, and close the ticket.
Even well-known benchmarks miss this:
- 42%. An agent that succeeds 75% of the time gets three runs in a row right only 0.75³ ≈ 42% of the time. Passing once in k tries (pass@k) is a very different promise from passing k times in a row (pass^k).
- 38%. On τ-bench’s airline tasks, an agent that does nothing at all passes 38%, because those tasks only check that the database wasn’t changed (Zhu et al., 2025).
- 24%. Adding missing unit tests to SWE-bench Verified changed the rank of 11 of 45 leaderboard entries (Yu et al., ACL 2025).
If a public, expert-reviewed benchmark can be passed by doing nothing, an internal eval written in an afternoon can be too.
The loop every change goes through
If testing an idea takes minutes, you try many and keep what works. If it takes a week, you try a few and guess. So the time from hypothesis to answer is the number we work hardest to shrink.
1. State a hypothesis, then version the change. Every change starts with a sentence: “The agent fails on X because of Y; changing Z should fix it.” The change becomes a separate, versioned candidate, so experiments never touch what’s running. The owner decides which levers the loop may pull: model, skills, tools, knowledge base and memory. The base prompt and code stay fixed, so every result traces back to one named lever. A cost saving only counts if quality holds.
2. Run it, never in production. Candidates run on three layers, each with a different blind spot:
| Layer | When | What’s in it | Blind spot |
| Datasets | Every change, in minutes | Golden scenarios, customer cases, captured real runs | Only the cases someone wrote |
| Benchmarks | Every serious candidate | Public suites such as SREGym, plus our own simulations | Still a simulation |
| Shadow runs | Before every promotion | The candidate works beside the production agent on real tasks but never acts | Slow, noisy, outcomes often unconfirmed |
Like slices of Swiss cheese, the holes don’t line up, so a failure has to slip through all three. Within each layer, we pick the cheapest test that can actually see the change. Replaying one model call is enough for a reworded skill. A new tool, model, knowledge source, or memory changes what the agent sees, so it always gets a full scenario run. A gain seen only on synthetic data stays provisional until real cases back it up.
3. Analyze every run, not just the ones someone had time for. An insights engine labels each run (task type, environment, tools used, outcome, cost) and looks for patterns across them: repeated tool calls, wasted context, an expensive model where a cheaper one would do, missing skills or knowledge. Each finding comes with the runs that show it, how often it happens and how confident we are. Some findings are blockers, like a missing integration, and the report says so. Engineers still read traces; it’s how we sanity-check the findings, and where many of the best ideas start.
4. Let the coach iterate, inside fixed rules. The coach is an agent that studies another agent. It reads the findings, the scores, the customer’s goals and every earlier attempt, including the failures. It builds a few variants and runs a quick test on the target cases plus must-pass cases, so fixing one failure can’t quietly break another. Before any experiment, we run the baseline several times to measure its natural noise; a gain only counts if it clears that bar. The coach cannot:
- change how it’s graded,
- choose or see its own final exam (the held-out cases),
- paste test answers into skills, or
- ship anything.
5. Compare, then promote. Quick rounds weed out weak ideas early. Only a full comparison on held-out cases moves a candidate forward, and only a person promotes it to the new baseline.
Everything rests on the grade
Every decision about a new version depends on how it was graded. A weak evaluator is worse than none: it points you the wrong way and makes you confident about it. So no single source of truth is trusted alone.
| Source | Covers | Strength | Limit |
| Automated graders (known-answer checks, LLM judges) | Every run, in minutes | The only way to grade at scale | Only as good as their calibration |
| People (pairwise comparisons, expert review) | A sample, where it matters | They define what good looks like | Slow, scarce, and experts disagree |
| Real outcomes (accepted or corrected answers, postmortems) | Cases that get confirmed | Ground truth | Arrives late, often never |
People calibrate the judges, and real outcomes check both. No LLM judge is trusted until it agrees with the customer’s experts, and agreement is re-measured whenever the model, rubric or case mix changes. When a judge and an expert disagree, we treat it as a bug in the judge and fix it with the same loop we use on the agent.
Two rules keep the numbers honest:
- Learning and measuring stay apart. The coach experiments on development cases, regression cases protect what already works, and held-out cases give the final verdict. Nobody tunes against them.
- Every comparison is pinned. It records the exact baseline, candidate, dataset and evaluator versions. The evaluator never changes mid-comparison, and a critical regression stays visible even when the average goes up.
Automated between three human decisions
Run those steps back to back, with a loop runner managing the budget and deciding what runs next, and you have the factory. The goal is that a customer can say “resolve more issues without increasing cost” and leave it running. People make three calls:
- Set the charter. The owner approves the goal, the limits, which levers the loop may pull, and the budget.
- Accept the evaluators. A domain expert reviews how grading works and where it disagrees with experts.
- Promote a version. The owner accepts or rejects it, with a reason the loop keeps for next time.
The loop stops when the goal is confirmed, the budget runs out, or rounds stop producing real gains. Either way, the owner gets a report: what improved, what regressed, the hypothesis behind each change, and what it cost. If nothing earned promotion, the report says so and the current version stays. More revisions isn’t progress.
Where we start
We’re proving the full loop on bounded benchmarks first, where ground truth is clean, and results are hard to argue with: an SRE agent on an incident benchmark such as ITBench, and a coding agent on SWE-bench. Two domains show which parts carry over and which need custom setup. Next, the same machinery moves to production-driven improvement, where live runs surface the opportunities, and to cost optimization at a reliable quality level.
What this means for teams running agents
- Agents that improve on your own work, continuously, not only when a big release ships.
- The trade-off you choose: more precision where mistakes are expensive, lower cost where volume is high, and no silent downgrades.
- Proof before production: how a new version performed on your real workload before it reaches you.
- The same rigor for your own agents: bring an agent built on our SDK and a goal, and get back a better version with the evidence behind it.
That last one matters most to us. The agents you build deserve the same evaluation, and the same human gates, as ours.
See judges, golden scenarios, and shadow runs in action in the Komodor Agentic Operations Platform demo.
