On 28 September, Anthropic published a playbook for designing evals and hillclimbing against them, and shipped two commands that put it to work. In Claude Code, /claude-api build-eval interviews you and builds an eval inside your codebase. /claude-api hillclimb then improves your app against it, one change per round, with a held-out split to catch overfitting.
It is a good piece of work, and the results are concrete. On an internal customer support benchmark, the hillclimb took a configuration from 78.6% to 90.5% on 14 held-out tickets the search never saw, at about one fifth of the cost.
We have been building toward the same goal for agents that operate production systems: agents that measurably improve, round after round, without anyone fooling themselves. We call our version the Evaluation Factory. Reading the two side by side is useful, because they agree on almost every principle and split on almost every constraint.
This post covers both: what we share, what changes when the agent’s job is to investigate and fix live infrastructure, and how the two fit together.
Where We Agree
Most of what makes an improvement loop trustworthy is the same, whatever the agent does. Anthropic’s playbook and our design converge on these:
| Principle | In the hillclimb | In the evaluation factory |
|---|---|---|
| Keep a set the optimizer never sees | Random train and test split; a patch that lifts train but not test is reverted | Development, regression, and held-out cases; held-out cases sit outside the coach’s selection |
| One attributable change per round | One patch per round, aimed at a root cause | One change, its hypothesis and its test |
| Know your noise before you act | Checks that score noise is smaller than the smallest gain you’d act on | Repeated baseline runs set a minimum detectable effect before any experiment |
| Never paste answers into the agent | Failure content never goes into the prompt | A skill must capture a reusable procedure, never a copied answer |
| Read transcripts to validate the grader | Claude grades a handful of cases and asks whether you’d score them differently | Judges are trusted only where they agree with people, and re-checked when anything changes |
| A flat result is a result | If the gain is within noise, it says so and recommends against merging | If nothing earned promotion, the report says so; more revisions is not progress |
| Cost is a legitimate goal | “Lower cost while performance holds” is a first-class objective | Cost improvements count only when the required quality holds |
If you are building evals for any agent, those seven rows are the part to get right first.
What Changes When the Agent Operates Production
The hillclimb is built around one app, one eval, and one session in your repo. That is the right shape for an inbox router or an API skill. An agent that investigates incidents and changes live infrastructure adds five constraints.
1. The eval set is never finished
Anthropic’s first principle is that eval tasks should mirror production, and its build-eval command samples production transcripts first. That gets you a strong eval at the moment you build it. In operations, the distribution moves under you: a new service ships, a dependency changes, an incident type appears that no one has seen.
So the factory treats the eval set as a living thing. Shadow runs put a candidate next to the production agent on real work, without acting. Customer corrections and linked postmortems confirm outcomes, and confirmed cases flow back into the datasets. The eval improves on the same loop as the agent.
2. Graders have to keep earning trust
The build-eval command validates the grader before the baseline runs. It checks whether the grader gives the same verdict twice, and it asks you whether you would have scored a few cases differently. That catches most broken graders at the start.
Over months, judges drift: a new model, a new rubric, or a new kind of case changes what they reward. A wrong grader is confidently wrong, and it points the whole loop the wrong way. Our design uses three sources of a grade, each checking the others. Automated graders cover every run. People calibrate them through pairwise comparisons and spot checks. Real outcomes- the accepted, edited, or rejected answers and corrected root causes- confirm both. Evaluators are versioned and fixed for the length of any comparison, and when a judge disagrees with an expert, that disagreement is treated as a hypothesis about the judge.
3. The path matters, not only the output
For an inbox router, the output is the result: the right queue, or not. For an agent with production access, two runs can reach the same root cause while one of them ran a risky command on the way. An average score hides the one unsafe run.
The factory grades every run on three things separately: the outcome, the path it took, and what it cost. A cheaper candidate that reaches the right answer by a riskier path does not get promoted on the strength of its average.
4. You can’t rerun production
Anthropic warns that leftover state from an earlier trial, a file or a git history, can hand an agent the answer. In operations, the state problem is bigger: the environment is a live cluster, and every rerun could change it.
The factory runs candidates in a synthetic world, in test environments, against replays of recorded runs, or against recorded responses, and never in production. It picks the cheapest test that can actually see the change. A single replayed model call is enough for some skill edits. Fork-and-continue restores a run to a chosen point and lets the candidate finish. Changes to tools, knowledge, memory or the model always get full-scenario tests, because they change what the agent sees. When a candidate takes a path the recordings never covered, the divergence is recorded, and the case runs live in a test environment or is dropped.
5. More than one person decides
The hillclimb runs with the developer in the loop and leaves the best version in their working tree. That works when one engineer owns one app.
An agent fleet has owners, reviewers and a security team, and some agents belong to teams that never wrote the eval. So the loop is bounded by three human decisions. An owner approves a charter: the goal, the limits, what the loop may change and the budget. A domain reviewer accepts the evaluators. An owner promotes a version, with a reason the loop keeps for later attempts. Everything between those points runs on its own, and nothing reaches production without a person’s approval.
At a Glance
The two are built for different jobs, and each is the right shape for its own.
| Claude Code hillclimb | Komodor evaluation factory | |
|---|---|---|
| Built for | One app or skill, in your codebase | A fleet of operations agents, in production |
| Where it runs | A Claude Code session, with your eval in the repo | The platform, round after round, within a charter and a budget |
| What it can change | System prompt, skills, tool descriptions, model and effort, harness code | Model, skills, tools, knowledge base, memory; base prompt and code fixed in the first version |
| Where cases come from | Production transcripts, tickets, hand-written and synthetic cases, built once | Golden scenarios, customer cases, captured runs and benchmarks, plus confirmed outcomes that keep arriving |
| How it’s graded | Code checks or an LLM judge, validated with you before the baseline | Code checks, judges and people on outcome, path and cost, recalibrated against real outcomes |
| Where candidates run | Your local harness | Synthetic world, test environments, replays and mocks; shadow runs next to production |
| Who decides | The developer running the session | An owner approves the charter and promotes; a reviewer accepts the evaluators |
| Status | Ships today in Claude Code | Tracing, judges, golden scenarios and shadow runs today; the automated loop is in development |
Where the Hillclimb Is the Right Tool
If you own one app or one skill and the code lives in your repo, start with the hillclimb. It is fast, it ships today, and it can reach parts of the agent our first version deliberately leaves alone: the system prompt and the harness code. For a skill’s trigger rate, a prompt audit or a model-and-effort sweep for cost, it is hard to beat.
A few ideas from the playbook are worth borrowing for any eval, ours included:
- Ask the grader twice. Run it twice on the same output and check whether the verdict changes. It is the cheapest grader test there is.
- Check for headroom before you climb. If the baseline already scores around 95%, aim the loop at cost or latency instead of quality.
- Don’t sample only today’s failures. Cases picked because the current model fails them measure that model’s blind spots, not what is hard about the job. Pick hard cases because a person can say why they are hard.
- Sort failures by cause when the score stalls. Their reflection step found a grader that contradicted the docs and a task that asked for something its grader didn’t check. Stalls are often the eval’s fault, not the agent’s.
Use Both: Tune in the Repo, Keep Improving in Production
These are two stages of one agent’s life, not two answers to one question.
- Build and tune in the repo. Write the agent with the Claude Agent SDK, build its first eval with build-eval, and hillclimb the prompt, skills and model until the held-out score stops moving.
- Bring it to the platform. Import the agent into the Komodor Agentic Operations Platform. It runs with the same scoped permissions, audit trail and approval gates as every other agent in the fleet, governed from the first run. You wrap it; you don’t rewrite it.
- Keep improving it on real work. Every run is traced and graded. Shadow runs test candidates next to production without acting, and confirmed outcomes keep the eval honest as the environment changes. A person still decides what ships.
The hillclimb gets an agent to a good first version fast. The factory is designed to keep it good after the infrastructure it works on has changed three times.
On status: tracing, judges, golden scenarios, and shadow runs are on the platform today. The automated loop described above, with the coach, the insights engine, and the loop runner, is what we are building now, starting with an SRE agent on an incident benchmark and a coding agent on SWE-bench.
Summary
Anthropic’s playbook is the clearest public guide we have seen to improving an agent without fooling yourself, and the principles carry over to any agent. What changes in operations is everything around them: an eval set that has to keep up with a moving environment, graders that have to keep earning trust, scores for the path as well as the answer, test environments that can’t touch production, and more than one person signing off.
Read how the loop works in The Evaluation Factory, or see judges, golden scenarios, and shadow runs in the platform demo.
