Self-Improving Agents: How to Turn Failures into Fixes Without Babysitting Prompts

Komodor Webinar — Recorded October 28, 2026

Speakers: Asaf and Dinor , from the Komodor AI Engineering Group 

Asaf Savich and Dinor Nagar show how to make a trusted but expensive AI agent cheaper to run without losing quality. Their method uses a teacher, student, evaluator and coach that improve the cheaper agent on their own and bring in an engineer only at the end. In a live demo, a Claude Haiku support agent climbed from 0.42 to 0.79 against a 0.85 Claude Sonnet teacher in three rounds, with only its prompt changed.

TL;DR: Breakdown

• Focus: How to cut an agent’s running cost, for example by moving it to a cheaper model, while keeping its results as good as before.
• Core Concepts: Four roles work in a loop: a teacher (the trusted reference agent), a student (the cheaper agent), a blind evaluator that scores and explains each answer, and a coach that proposes revisions, with a human review only at the end.
• Includes: A live demo built on a public repo, a three-step path to trust (historical replay, shadow runs, controlled A/B), and six failure modes with how to avoid each.
• The demo repo that you can run on your own, is publically available at https://github.com/komodorio/hello-world-continual-learning.

Key Takeaways from the Webinar:

• Organizations waste expensive engineering time by manually “babysitting” prompts to make cheaper AI models perform complex tasks reliably.
• The teacher–student setup gives a clear target: the cheaper agent only has to match the trusted agent’s accuracy, not be perfect.
• An “Evaluator” acts as a blind judge, scoring the cheaper “Student” agent against the Teacher’s standard and explaining the reasoning behind any failures without revealing which agent wrote which response.
• With the human moved from every iteration to one review at the end, the loop runs on its own; in the demo the student closed most of the gap in three rounds, and no engineer wrote any of the revisions.
• The loop needs guardrails to be worth running: clear correctness requirements, a held-out test set, a coach that remembers past changes, a budget, freedom to change more than the prompt, and a staged rollout before anything ships.

Webinar Transcript (edited for clarity, with filler words removed)

All right, let’s get started. Thank you all for joining. Today’s webinar is “Self-Improving Agents: Turn Failures into Fixes Without Babysitting Prompts.” We’ll cover a lot today, so bear with me.
Some housekeeping: if you have questions, use the Zoom Q&A and we’ll get to them at the end. Yes, this webinar is being recorded, and we’ll share the recording with you within 48 hours. We’ll also show a demo, and we’ll share the link to the demo repo in the slides and in the follow-up email.

We’ll cover the actual problem, how teams today end up babysitting agents to improve their scores, and we’ll show our teacher, student, evaluator and coach model live.
Now let me introduce our speakers. With me today is Asaf Savich, AI Engineering Group Manager at Komodor. He’s been told that people stay awake for 70% of his talks. He speaks at cloud-native conferences such as KCDs. Also with us is Dinor Nagar, AI Engineer and Researcher, who holds a slight lead over Claude in Komodor commits. With that, I’ll hand it over to Asaf to start the webinar.

Why improving agents is hard

Asaf:
Hello, everyone. I’m excited you could all join, and I’m really happy about this webinar. We run quite a few of these on many different topics, but improving agents, and making them run the way we want and as efficiently as we want, is a very exciting topic. I also think it’s the next logical step.
We’ve implemented agents. We’ve run quite a few of them. We know basically how they work, and we have some trust in them. The next logical phase is gaining trust that they work the way we want them to, and making them run as well as we want them to.
Improving agents covers many sub-topics and categories, and I think we’ll have enough webinars to cover all of them. In this one, Dinor and I want to focus on one category: you have an agent that works quite well, but it costs a lot. You want the same agent to produce the same results, just at a lower cost. That’s the optimization we’re focusing on today. At the end, we’ll touch on other methods, but not in as much depth.
Let’s make sure we all understand the use case, what we’re trying to improve, and why it’s a problem.
Phase one: we have a trusted agent. It works. I’ve seen many of its runs and gotten feedback on it. That doesn’t mean it’s perfect, that it handles every use case, or that it has the latency I want. But it works, and I have some trust in it.
The problem is that it costs a lot, so I want to try something different. For example, I want to try a cheaper model. I’m working with Sonnet and want to try Kimi, or I’m working with Fable and want to try Sonnet. I want a model that isn’t as advanced, as smart or, of course, as expensive as the current one.

So I try the cheaper model and, no surprise, quality drops. Now I have a problem. What should I do?

What we see many of our customers do today is hand it to an engineer and ask them to make the agent work as well as it did before the change. The engineer can try a lot of tricks: change the prompt, adjust the setup, try again, maybe add some skills. It’s all trial and error. Even when they reach a point where the change feels okay, it’s been tested on a very small subset of the use cases the agent handles. The fact that the three or four cases you tested work doesn’t mean the 100 different cases the agent used to handle still work.

So improving an agent is not easy. It takes a lot of time, a lot of money and a lot of understanding. In the organizations we work with, this job usually goes to the most trusted, smartest and most capable engineers. So it’s not only a lot of time, it’s the most expensive time you have.

We tried to help, both for our own sake, because we have many agents we want to improve and make cheaper, and for our customers who use them. We’ve spent a lot of time and effort figuring out how to make this work, and today we’ll share how we made it better.
Instead of that super smart, capable engineer working on this, we want them to be as little involved as possible. We want them to have trust and confidence that the agent really costs less and delivers the same value, without spending a lot of time on it and without the trial and error we just described.

We want to give agents the freedom to make the modifications themselves, with the proper guardrails, so everything is safe and the result is easy to digest, not something you’re afraid of changing. There are different ways to change an agent: you can change its model, prompt, tools, harness, code and so on.

Learning from the classroom

We started thinking: how can we make agents learn and improve? Learning isn’t new. In learning in general, not just agent learning, you have a task and you try to complete it. It might be a math problem or anything else.
Say I’m in a math class and I submit my answers. Someone checks them and says these are wrong and these are right. So I know which three or four answers I need to improve. I study the subject and try to think about it differently.
I do a kind of self-optimization. I understand what I did wrong in those answers and try to generalize, so I’m not only able to solve the specific tasks I failed, but understand the concepts well enough to ace any question thrown at me. These are the steps and mechanisms we want to follow so that agents can improve themselves.

The four roles: teacher, student, evaluator, coach

We’ve come up with a mechanism that works quite well, and we use terminology from the school system to describe it.
Before we talk about the teacher, the student and the rest, here’s the key principle at the bottom of the slide: the goal is for the second agent, the one with the cheaper model, to be as good as the main one. In this specific focus, its goal is not to be the best agent in the world or to ace every single question. It’s to be just as good at a lower price. That’s what we’re aiming for.

We have four entities. The teacher sets the bar. It’s the agent we trust: it has handled many different tasks, and we know it answers them quite well. The student is the agent trying to be as good as the teacher. It’s the main role here and the entity we’re trying to optimize in this session. The evaluator, or grader, judges whether the student and the teacher did a good job. We’ll talk about it in a second. And the coach is like a private tutor: it takes my failures and teaches me so I can get better.
Let’s start with the teacher. The teacher is my current main agent, the one I rely on. It does a pretty good job because it uses a very expensive model. As we said, the teacher sets the standard, and the mechanism aims to be as good as the teacher.
It’s also important to say, and I’ll say it a few more times, that the teacher is not perfect. There are downsides to what we’re presenting today, and Dinor will cover them later because it’s important to understand them. But we want to be as good as the teacher. Just as important, we need to compare apples to apples: the student and the teacher must both get the same input.

So what is the student? Usually, though not always, it’s a cheaper model. We want cheaper results with the same accuracy. But it can be other things too. You can take some logic out of the agent and make it deterministic, like a script. You can change the model. Or you can do other things to make the run cheaper or limit it in some way. The student is the agent we want to improve.

How do we make that work? We have the evaluator, the grader, which scores both. Say I have a task. Both the teacher and the student run it, and then the evaluator grades both results without knowing which is which. This is very important. We don’t want the evaluator to be biased. Because it knows about the mechanism, it might infer that the teacher’s result is probably better. So it must grade blindly, with no indication of which agent produced which output.

The evaluator doesn’t just say “this run is a 70.” It explains why it gave a 70 and not a 90. Maybe the answer found the root cause, but the suggested remediation was wrong. Maybe it didn’t take the user’s tone into account. The evaluator must not only give the score but also explain it. That’s very important for the coach.

The last agent in this mechanism is the coach. It’s the one that holds the brain. It usually uses a stronger, more frontier model, like Fable. It doesn’t run often, but it does the most important job here: improving the student. To do that, the coach has to be very smart and understanding. Usually we even want the coach to be smarter than the teacher.

The coach reads the failures from the evaluator, and it never sees an answer key. It doesn’t care what the correct answer is. It only cares how far the student is from the teacher, because the goal is to bring the student up to the teacher’s level. So it doesn’t really care whether an answer was correct, but it does care when the teacher was right and the student was wrong. Once it understands that, it proposes a new revision: the next change that should make the agent better.

Those are the four entities.

How the loop runs

What I described is basically this diagram. In phase one, the teacher and the student both run on a task. The loop doesn’t care what the task is. In the demo you’ll see in a few minutes, it’s a support ticket. Both the teacher and the student are support bots whose goal is to keep the customer satisfied and make sure they get what they came for.

Both run, then the evaluation happens, the coach proposes a change, and everything is re-run.

Originally, we had a human review at this stage. We’d say: this is the change we want to make, for example a prompt change the coach suggested to improve the student. A human would review it, and only if it wasn’t good enough would we go back to step one.
We wanted to do better. Instead of a human in every cycle of the loop, first change, second change, third, we wanted the human out of the loop. So we changed it to this: the re-run goes back to step one, and only when the loop finishes does that super capable engineer get a result, once the coach says it’s good enough.

The loop can run many cycles. The first cycle changes the prompt. The second adds a skill. The third removes two tools that were never called. The fourth does something else. We used to have a human review in each of those cycles. Then we realized the human doesn’t have much to say, because there’s nothing good or bad to say about a proposed revision until it has run again and been evaluated again.

So we moved the human out of the loop. Now the human comes in only when the agent says: “These are the four changes I made. It’s good enough now. I sign off on it, and it’s up to you to review it, challenge me, or approve it.” Once it’s approved, you promote that revision, and you can add the examples it was tested on to your trusted examples. Those then improve your next cycle of improvement.

The demo: a support desk with four traps

Dinor Nagar
So far, we’ve covered all the components of the loop and how they connect: the student, the teacher, the evaluator and the coach. Now we want to show how it works on an actual use case, with a demo we built beforehand.
As Asaf mentioned, the scenario is a support agent that answers my customers’ questions. For the demo, we limited the data to four tickets: refund, multi-request, missing feature and compensation.

You’ll also see a column called “the trap.” The trap is what the agent needs to understand about the ticket to reach the right solution. Keep this in mind, because it’s central to the loop: the agent has to figure out what’s missing, and the coach’s role is to fill in that missing information.

A few details about the demo. The repo is public, so you can run it yourself, and there’s a README that explains how to get it working.

Some additional details that matter for the experiment. The teacher, evaluator and coach run on Claude Sonnet, our strong model here. We don’t change their model during the experiment, because we want to keep the same baseline when we compare against the student. The student runs on Claude Haiku, the weaker model in this case.

We only change the student’s prompt. As we said, we could also change the code or the harness, but we kept the experiment scoped to the prompt. You’ll also see that we limit it to three rounds. We define success as the student’s accuracy coming within 0.1 of the teacher’s.

Now I’ll share my screen. What you see here is the experiment. I’ve prepared a video showing each iteration of the loop, and then we’ll dive into the UI itself and I’ll explain all the components.
Very briefly: on the left are all the data points we talked about. In the loop view, you see each component and the model it uses.

We’ve started the loop. On the right you can see the evaluation scores. We have four tickets, and in a moment we’ll see the first iteration’s accuracy. The teacher got 85% and the student got 42%, which is relatively low. That makes sense, because it’s the first iteration and we haven’t changed anything yet.

This graph shows the student’s improvement compared to the teacher.

Now we see the second round: an increase from 42% to 66%. We can see which tickets improved and which didn’t, which helps us understand what happens during each iteration.

The third round brings a further increase, and the student lands at 79%, relatively close to the teacher. Keep in mind that we started from a weaker, cheaper model, Haiku, and changed only the prompt to bring it as close as possible to a teacher that uses a
stronger model.

Now let’s look at the actual UI. On the left is the improvement graph we talked about. We can also see the diff of the prompt itself: what the coach decided to change and what information it added to the student’s prompt to make it more accurate. When you run it on your own machine, you can analyze what happened and visualize the whole experiment.

Asaf Savich
It’s important to say that this demo is quite straightforward, because it only shows improvements: the first cycle improves, the second improves, the third improves. In the real world, we often see drops, and then the loop goes back. As we said, the coach is a very smart agent and can handle complicated situations. We see this a lot: the coach makes a revision that lowers the score a bit, and then it fixes it, rolls it back, or uses the failure to decide on the next change.
The most impressive thing is that this is what the engineer gets when they wake up in the morning. Everything up to the approval is done fully autonomously. As an engineer, that makes my life much easier, and I trust it much more than if I’d made all those changes manually and tried to understand them myself.

Dinor Nagar

One more thing to add: our dataset was very small. Four tickets aren’t enough for a real production system. What we see with our customers is that you need at least tens, or even hundreds, of tickets to get a good distribution of what the customer’s data looks like. Keep that in mind as well.

Going back to the slides. To recap: we started with a student using a weaker model than the teacher, with a huge gap between them. As the learning iterations continued with the coach, we successfully brought its accuracy close to the teacher’s, which was our goal.
Here’s a short recap of what changed. The first version gave the student the same prompt as the teacher and only switched the model to Haiku. Then we ran two more iterations that made changes to the prompt.

The road to self-improving agents

Asaf Savich

I see there are some questions in the chat. We’ll answer them shortly.

What Dinor just showed is a single experiment. We had one agent we focused on improving, the support agent in our organization, and the trigger was manual. We said: let’s run a loop, let’s set up the teacher, student, evaluator and coach, let’s make it work, and I’ll review it at the end. That’s nice. It ran autonomously, I reviewed it at the end, and say I approved it and it’s now in production.

But this is only the beginning. We believe the real improvement is for organizations with many agents and many stakeholders, doing very complex tasks on very large datasets. There, improvement has to be continuous. It won’t be a nice-to-have; in the near future it will be a must.

We envision a machine whose job is to understand which agents have optimization opportunities. The agents decide which agents should be optimized, and I, as the engineer, only approve the changes at the end. If I have enough trust in that machinery, which isn’t the case yet, and the agent isn’t very sensitive, meaning it doesn’t touch PII, production or anything like that, the entire process could even run autonomously. That’s a big leap, but we think it’s where things are heading over the next few months or years.

Earning trust before you promote a change

How do we gain trust? That’s the big question. We have an agent, and we want to trust its results, its predictability, how well it works and how our users react to it. There are a few mechanisms for gaining that trust.

The first is historical replay, which carries almost no risk. I take past runs of the agent where I know what the expected result should be, and I rerun them again and again until I have some trust. Say I made a prompt change or a model change and want to see that it works. I can run it on past cases with almost no risk and get quick value, but not amazing value. The fact that it worked on previous runs doesn’t mean it will work as well on future ones.

The second is a shadow run. Say the support agent is running in production. I deploy another support agent, call it v2, and both run in production. All users reach the main support agent, but without knowing it, another agent works under the hood on the same task. Users are only exposed to the main agent, while v2 runs in the background. This way, I can validate on real production use cases that the agent is doing a good job. After a few hundred thousand sessions, I know for a fact that it’s good enough, or even better than the previous agent, and I’m confident enough to promote it.

The next step, which does expose users, is A/B testing. I’ve run it on past incidents, I’ve run it in shadow for some time, and I have enough confidence. The next logical phase is to make it live, but not 100% live. Start with a split, 80/20 or 70/30, it doesn’t matter. You see results, but the risk is contained, because if something goes wrong, only a share of the requests go to the so-called bad agent. That gives me full control. If it does a good job after running in production for a week on 20–30% of the tasks, I can promote it and feel safe.

At each of these steps, a human is the gate. A human decides when to move from a dataset to shadow, and from shadow to A/B testing. That’s how we see it working among our customers today.

Six failure modes to watch for

Dinor Nagar

It’s also important to talk about what can fail in the process. We saw a successful demo, and you might think: great, I’ll go to production, run the loop and everything will be fine. Unfortunately, in our experience we ran into many issues along the way, and you should keep them in mind when you try this on your own data. We’ll cover six main failures you may encounter.

1. Wrong optimization target. Remember that if we don’t clearly define what a good answer is, the evaluator can get it wrong. You can end up with a student that writes a fluent, well-styled, very long answer that is 100% incorrect. If you haven’t defined what a correct answer is and what you expect it to look like, the evaluator might give it a high score. That’s very bad, because we’d see an improvement while the student isn’t really improving. This is a critical point.

2. Dataset overfitting. Overfitting isn’t new; it’s a known issue from the machine learning world. For those not familiar with it: I focus on my training data and get very good accuracy, then I deploy the agent to production and accuracy suddenly drops, and I ask myself what happened. That’s overfitting. To make sure your training isn’t overfitting, keep another subset of data on the side, called a test set or validation set. In every training round, also calculate accuracy on the test set. That way you can make sure training and test accuracy improve together.

3. Inherited teacher errors. Just like in real life, teachers make mistakes. When we train the student according to the teacher, we might inherit the teacher’s errors. It’s very hard to avoid this completely, but you can reduce it. If you validate the teacher against well-defined requirements, meaning what a good answer is and how it should answer specific questions, you can be much more confident in the teacher and make sure it produces fewer inaccuracies.

4. Iteration degradation. This one is crucial. We run many rounds, say up to round five. If the coach doesn’t remember what it changed before, it might break its own changes. For example, in round four it adds something to the prompt, and in round five, not knowing what it did before, it deletes it. Then we get an endless loop where accuracy never converges, which is very bad. To prevent this, the coach needs to understand and remember its previous changes so it doesn’t overwrite them.

5. Process economics ignored. When we work with agents, we tend to forget about cost. But calling LLMs costs money, sometimes a lot, especially with a lot of data running in a loop. You might end up with a $2,000 loop that only saved $20 on your agent, and sometimes that’s not worth it. So it’s important to set a budget for the loop itself, so you’re fully aware of your costs and the loop doesn’t go beyond what you defined up front.

6. Wrong component changed. In the demo, we only changed the prompt. But sometimes changing the prompt isn’t the right move, and it will only get you to a certain point. Say we want to reduce cost on an agent that uses a very high-end model, but we say we don’t want to change the model. There’s a limit to how much prompt changes alone can reduce cost. If we don’t also let the coach change the model, we might not reach our goal. So it’s important to define what kinds of changes the coach can make: the model, tools, skills and so on.

Wrap-up: from Klaudia to the Komodor Agentic Operations Platform

Asaf Savich
We’re at the end of the webinar. One small thing I forgot to mention earlier: as Dinor said, this can get pricey. Once you give agents the autonomy to run, they can run for a long time and cost a lot of money. So when you define a loop, you can set its limits: a time limit, a budget limit, or an accuracy target, so it stops once it reaches a certain accuracy or a certain cost. We have a lot of tooling around that.
In general, here’s how we got here. For a long time we’ve worked on Klaudia, for those of you who know her, our AI SRE. Klaudia is a super smart, sophisticated agent that can solve all sorts of problems. While building Klaudia, we put a lot of focus on optimizing her. We got to a point where she does a good job relatively quickly.

Then we had to improve her: make her cost less, make her smarter, make her aware of everything that’s happening, help her understand each customer’s knowledge, and much more. Those improvements meant optimizing her on a daily basis. There was never a point when Klaudia wasn’t running four, five or six experiments in parallel.

That taught us the depth of agent optimization and how to do it. It started with our own needs, just optimizing our own agent. But we realized this pain isn’t unique to Komodor. That led us to build and launch the Komodor Agentic Operations Platform, which is now widely used.

It has several pillars I can explain briefly, but mainly the optimization pillar. The optimization pillar includes this loop, but not only that. There are also golden scenarios: scenarios where you understand the use case your agent is trying to solve well. You give the agent a live environment, run it, and get very good scores with very little effort. It’s a very good shift-left method for agent optimization. There are also the datasets we talked about briefly, the loop we just discussed, and many other practices.
We started this webinar by saying there are many methods for optimizing agents. Today we focused on a mechanism that optimizes agents for cost. In other sessions we’ll talk a lot about making them more accurate, which is as interesting as saving cost, maybe even more. Our platform fully supports this, and we use it for our own purposes. Seeing how customers use it is really impressive. It’s an amazing time to be a builder, creating agents and tackling these new-world problems. Anyone who wants to can sign up for a free trial and give it a try.

Host:
As we mentioned earlier, you can run the demo Dinor showed yourself; we’ll drop the link in the chat. And as Asaf said, you can use the free trial. Our next webinar is in about three weeks, in November, and it will cover more of the optimization methods Asaf mentioned. There’s a lot of progress and cool things happening behind the scenes in the Komodor Agentic Operations Platform, and we’ll talk much more about them in the coming months, so stay tuned!

Q&A

Now I’ll hit you with the questions. You’re the brains, I’m just the pretty face. The first question is from Shimona: how many samples are considered for each round of improvement?
Dinor Nagar

We’ve touched on this, but it’s worth repeating. In the demo we had four, which is far too few. From what we’ve seen in real life, you need a few hundred cases in your dataset, and the more the better. Keep in mind that getting this data is very expensive and sometimes very hard: collecting it from customers, aligning it and cleaning it to make sure it’s correct. But if you want a number, I’d say start with a few hundred, and more is better.

Asaf Savich
I can take the next one. The question is: the code for the teacher and the student is identical except for the model used, correct? And do they run in parallel, or was the teacher evaluated separately?
Good question, and it’s important to make this clear. In each cycle, we want to evaluate one change, whether it’s a prompt change or a tool change. That way you know whether your change was good or not.

In general, the code stays the same if you made a prompt change, and it stays the same if you made a model change. But a code change is also a valid change the coach can suggest. In a specific case, the coach might suggest changing the code, or even changing the harness, for example whether Claude Code or ADK is the right approach for the model you’re working with. So generally you’re using the same code, though the code can also change if the coach suggests it.

And yes, the student and the teacher, the agent that sets the standard and the one we’re trying to evolve, run in parallel. Both get the same requests, and then the evaluator agent evaluates each of them separately.

Another question: how can we know the coach is good in the first place?
Dinor Nagar

Remember that the coach’s main goal is to suggest changes that help the student get better. The coach is also an agent, so how do we make it better? The main thing is to check whether the coach gets the student to converge toward the teacher, and as fast as possible. If we’ve run, say, 50 or 60 rounds and the student doesn’t converge, that’s an indication the coach may be suggesting bad changes, and we need to investigate. But when we see convergence, like in the demo, and the student reaches our goal, that’s an indication the coach is good and working as planned.

Host
Any more questions? No? All right. Thank you so much, Dinor and Asaf. This was a great webinar. If you have any other questions, feel free to reach out to the Komodor team; you’ll find all the links below. Thank you all for joining, and we’ll see you next month. Bye-bye.

FAQ

Q:How much data do I need to run a teacher–student loop in production?

A:The demo used four support tickets only to keep things simple. For a real agent, start with a few hundred cases; more is better. Also keep a separate held-out test set and score it every round, so you can tell real improvement from overfitting.
Q:How do I know the coach is giving good advice?

A:Watch whether the student converges toward the teacher’s score, and how quickly. If many rounds go by without convergence, the coach is probably proposing poor changes and needs a closer look. It also helps if the coach can see its earlier revisions and their scores, so it doesn’t undo its own fixes.

Q:Can I run the demo myself?

A:Yes. The repo is at github.com/komodorio/hello-world-continual-learning and is MIT licensed. It needs Python 3.11+, uv and an Anthropic API key. By default, Claude Sonnet runs the teacher, evaluator and coach, and Claude Haiku runs the student. Only the student’s prompt changes, and the run stops when the student is within 0.1 of the teacher or after three rounds. Each round makes about 18 model calls.