Production-Grade AI SRE vs. ‘It Works on my Laptop’: What’s Missing & How to Add It

AI SRE on Your Laptop: What’s Missing and How to Add It

Komodor Webinar — Recorded July 28, 2026 Speaker: Andrei, Open Source Leader at Komodor 

In this webinar, Andrei (Open Source Leader, Komodor) breaks down what separates a laptop AI SRE prototype from a system an organization can actually run and trust. Drawing on his experience building Klaudia, Komodor’s AI SRE agent, Andrei traces the build-out step by step: removing the human operator, replacing interactive chat with an API and webhooks, layering in identity and RBAC so agents never run on personal credentials, and persisting sessions so results compound across a team instead of staying trapped on one laptop. He also covers what it takes to actually know whether a change helped, meaning LLM-as-judge evaluation and benchmark sets built from reproduced incidents, not guesswork. The session closes with audience Q&A on prompt injection defenses, who judges the judge, and what Andrei would build differently next time.

TL;DR: Why “AI SRE vs. It Works on My Laptop” matters:

• A laptop agent looks like progress but doesn’t scale. No operator means no one to trigger it, review it, or catch it when it fails.
• Getting to production means real infrastructure: an orchestrator, an API in place of interactive chat, and an identity/RBAC layer so agents run on scoped, revocable access – not someone’s personal credentials.
• Persisted, shareable sessions turn one person’s win into an organization’s, so a colleague can pick up an agent’s session and ask the next question.
• Cost has to be tracked from day one. Unmonitored agents burn tokens fast and silently.
• You can’t tell if a change made things better or worse without real evaluation. LLM-as-judge scoring and benchmark sets built from reproduced incidents, not intuition.
• Building this yourself is a real, ongoing engineering commitment, most companies are better off not reinventing it from scratch.

Key Takeaways from the Webinar:

• A working laptop agent creates a false sense of “solved”. Production readiness is an entirely different engineering problem.
• Removing the human operator requires an orchestrator, an API-first interface, and identity/RBAC, not just more compute.
• Persisted, shareable sessions are what let an agent’s value compound across a team instead of staying locked to one person’s laptop.
• Rigorous evaluation, including LLM-as-judge scoring and benchmark sets from reproduced real incidents is the only reliable way to know if a change actually helped.
• Cost-awareness and monitoring need to be built in from the start, not bolted on after the fact.
• Building and maintaining a production-grade AI SRE agent is a substantial, ongoing commitment, which is why Komodor built Klaudia as a product instead of something every team has to reinvent.

Webinar Transcript

Nikki: Welcome to the Komodor webinar series. We’re glad you could join us today for “AI SRE vs. ‘It Works on My Laptop’: What’s Missing and How to Add It.” We’re thrilled to have Andrei here with us — he’s the open-source leader here at Komodor, with two decades of industry experience, and has spoken at conferences all over the world. If you have questions, drop them in the chat and we’ll open the floor at the end. Andrei, take it away.

Andrei: Thanks, Nikki. I’ll share my screen — if you have questions during the presentation, throw them in the chat and I’ll get to them at the end with Nikki’s help.

[A brief screen-share issue followed — Andrei’s Linux laptop didn’t cooperate at first. “It worked on the laptop, why isn’t it working in production?” got a laugh from the room before he reconnected and started properly.]

Thanks for your patience, everyone. Here we go.

My name is Andrei, and I’ve been at Komodor for almost four years now. I’ve been through the process of building our AI SRE, and like everyone else, I’m living through this AI transformation — which is an interesting time to be an engineer. The transformation isn’t over, but we can already share some conclusions and experiences from living through it.

These days we all have pretty good AI on our laptops, and SRE is probably one of the best domains to try to automate using modern AI technology. I’ll share what I learned while building our AI SRE. Everyone starts today from the same place — a local, laptop-based setup — so I’ll assume we start there, and try to build up to something more production-ready, at least in theory. Then I’ll take your questions.

What Does “AI SRE” Mean?

I’m approaching this as a theoretical breakdown of what we learned. AI SRE, as an activity, breaks into a few use cases:

Incident response. When something happens and there’s time pressure, this is what you want to automate first. You want root cause analysis, you want to find the fix, and you want to carry that fix all the way through to recovery. This is the classic on-call-through-the-night scenario — the most typical and usually the first use case people go after.

Self-deployment and reconfiguration. When you want to make sure a change to your infrastructure or configuration is safe to proceed with. LLMs are genuinely good here — their encyclopedic knowledge of technology and their ability to reason about it is where you get real value.

Cost and resource optimization. We found that if you want real impact on SRE life, it pays to start early here — looking at resource optimization and financial outcomes, identifying underused resources, eliminating them, and saving real money. Nothing is better than AI that pays for itself through hard-dollar savings.

Those are roughly the three use cases we keep in mind. Your situation might surface others — it’s all competitive.

AI SRE on Your Laptop

What does it mean to run AI SRE on your laptop? You take a general-purpose agent harness — Claude Code, Codex, OpenCode — dozens of these exist. You give it your CLI tools, your credentials, maybe some MCP services, and you ask it to do the thing. It’ll do it — maybe slower, maybe needing some nudging, but this is the beauty of modern AI. It’s smart enough to get through, though not 100%; you can’t leave it unattended. There’s a lot of handholding and supervision required.

You can automate your typical tasks into agentic skills. New skills get published weekly — LinkedIn is full of stories of SREs automating their work, where what took hours now takes five minutes. If you look for open-source skill packs, they look like repositories of these skills. Whatever the author put into them, you can generally trust they put real effort and knowledge in. If you have nothing else, this is a good baseline.

The catch: it creates an illusion of ease, and it’s easy to assume you can scale that same setup beyond your laptop. That’s exactly what today is about.

The laptop diagram, roughly: an agent harness with skills built in, some tools like the AWS CLI and kubectl, your personal credentials, and occasionally outbound calls to internal MCP services (GitHub, Datadog, etc.).

What’s wrong with the laptop? You can push through the problems by nudging the agent until it concludes, but it has limits — you can only run as many agents as you have people and their attention. If you want to scale, you have to remove the operator. How do machines trigger it without a human copy-pasting incident messages into the agent? When it fails to analyze something, how do you troubleshoot that failure and close the improvement loop? How do you tell whether a change to your skills and setup made things better or worse — not just for the one case you reviewed, but broadly? How do you share not just the agent but its outcomes with your colleagues? And finally, how do you control the cost of the thing — because generic coding-agent harnesses are bloated; they’re built for the general case and burn a lot of tokens on things you don’t actually need for AI SRE.

Building Up From the Laptop

Here’s how we gradually took the laptop setup toward something more production-grade. (Andrei worked through an evolving architecture diagram, adding one component at a time.)

1. Give it a home. Move it off your laptop and onto a server, so closing your laptop doesn’t stop the work. Same harness, same setup — just hosted somewhere persistent.
2. Scale it. A one-to-one ratio between agents and humans defeats the purpose — we built these agents to delegate far more work than we can process ourselves. That means detaching AI from a human driver and introducing an agent orchestrator that spins up new agents when work arrives and shuts them down to save infrastructure cost when idle. Most agent work is I/O-bound, not CPU-bound — it’s mostly waiting on tool calls or LLM responses — so you don’t need much compute to run dozens or hundreds of agents. (Coding agents are the CPU-hungry exception, because of everything running in the background.)
3. Replace interactive chat with an HTTP API. Interactive chat assumes a human driver. Remove the human, and the interactivity stops making sense. Instead, you want an API — synchronous (start and wait) or, more commonly, asynchronous (start a session, poll for completion) — plus webhooks, so something like Datadog can trigger the agent directly. This is why interactive agent harnesses like Claude Code lose their relevance in this context. Komodor actually built its own agent harness two years ago, before many of today’s tools existed — and in 90%+ of cases, no human is needed to answer a follow-up question. You get the root cause, and that’s it.
4. Stop using personal credentials. Security matters. Agents need to either use the acting user’s identity (propagated through the system) or operate under a scoped service identity. This is where an identity/RBAC gateway comes in — it converts user credentials into infrastructure access at a reduced privilege level (e.g., read-only), so the agent can’t cause real harm even if it hallucinates or gets prompt-injected. This is one of the biggest gaps versus the laptop setup, where you’re constantly worried about mistakes. Decoupling the agent’s actions from personal credentials — via a proper identity and RBAC layer — is what separates something that scales to hundreds of sessions a day from something you fundamentally don’t trust.
5. Add the chat interface back — but for follow-up, not for driving. It’s rare that you’ll be happy with every result. You need a way to revive a session and ask “why did you do that?” — this is one of the most valuable tools for closing the improvement loop. Early on with Klaudia, we’d ask this constantly, and the answer was often “you didn’t give me the Kubernetes events” or “you didn’t give me the pod logs” — which directly improved what data we fed it.
6. Persist everything. Store inputs and results so sessions can be revisited and shared — turning something personal into something organizational. The most common pattern: a link to a session gets shared in Slack or attached to a Jira ticket, so a colleague can pick up from where you left off, or ask a follow-up question themselves. For debugging, you also want the full transcript — the tool calls, the LLM responses — and in more advanced setups, traces and KPIs like cost and latency.

Internally, Komodor has a system for reviewing every session transcript — what the LLM asked for, what tools returned, where the gaps were. That said, you shouldn’t expect to spend much time there day-to-day; a human can’t keep pace with agents running at scale. You mainly need it for the rare hallucination, or when a tool call silently breaks.

We also built our own tracing — off-the-shelf LLM tracing tools didn’t work well for us, so we wrote something slim that shows exactly what happened and in what sequence: parallelization, tool-call performance, time spent in intermediate reasoning versus final response, and total duration.

7. Treat it as a real service. Once you’ve added all of the above, this needs proper maintenance: code in a Git repository, a CI pipeline, and serious monitoring — cost data, alerts when agents start failing. We’ve had cases where a test environment ran out of credits and every agent quietly started saying “I need more budget” — you need to be watching for that. We use Datadog extensively internally to track failing tools, request intensity, latency, and exceptions in production.

Cost tracking matters here too — it’s easy for agents to burn tokens non-stop, so build cost-awareness in from day one; surprises are unpleasant.

8. Measure whether changes actually help. About six months into building Klaudia, we realized we couldn’t tell whether a given change made things better or worse. On a laptop, this is easy to eyeball. In production, at scale, it isn’t. When we talk to customers building their own agents and ask how they measure improvement, the honest answer is usually: they don’t.

Asking users for feedback (thumbs up/down) doesn’t work well in practice — it’s optional, and people don’t bother. So we built our own LLM-as-judge evaluation. It’s not as hard as it sounds once you know the pitfalls. The basic idea: take the output of an agent session, give it to another LLM along with grading guidance, and have it score the session.

The hard part in practice: naive LLM-as-judge tends to bucket scores (giving 30, 70, 80, but never 38, 55, or 73) because it reasons in language concepts, not numeric ones. The best sign that your grading instructions are working is that the score distribution looks roughly Gaussian across many sessions — spikes mean the LLM is defaulting to two or three buckets instead of a real range. The fix is to give it multi-dimensional criteria: did it provide evidence for its conclusion? Did it stop at the symptom level, or trace through multiple layers of the architecture to the actual root cause? Did it correctly identify the change that triggered the incident? Weight each dimension, and you get much more useful scoring.

This evaluation capability is the foundation for benchmarking. We use it, for example, to compare model versions — a newer model might win by 2.5% on quality but be slower and more expensive, and you have to decide whether 1% more quality is worth 30% more cost. We also track statistical saturation — running enough trials that score fluctuations plateau — before we consider a comparison conclusive.

This, in turn, feeds into the biggest piece of work: building actual benchmark sets — infrastructure for measuring agent performance predictively, not just after the fact. The hard part specific to AI SRE is that you have to reproduce real incidents: synthetic data sets, reproducible failure scripts, validation that the right kind of breakage actually happened. It’s a lot of hidden work, but it buys you the ability to answer questions like “what happens if I swap Claude Code for Codex?” or “how do we compare to a competing AI SRE tool?” with a real, repeatable measurement — not a guess.

Closing Thoughts

We started from AI SRE on a laptop and gradually built the pieces that turn a personal success into a company-wide one. It’s not trivial, but it can be done gradually.

There’s more we didn’t get into today — circuit breakers, an LLM gateway, an MCP gateway (a pattern I like, because MCP sprawl is real — too much tool surface without one), a notifications gateway, Slack integration (which is its own complicated beast). Open source exists for a lot of these building blocks, but open source gives you the code, not the expertise — you only get that by actually building and running it.

Building AI SRE isn’t most companies’ core business, and most companies are better off spending resources on their actual business. Be aware of what you’re signing up for if the success of a laptop prototype tempts you to think “AI solves this for us, let’s go.” We know this because building AI SRE is Komodor’s business — we’ve historically automated SRE work, we build this platform for customers, and this is how we learned everything I shared today.

Q&A

H: Who validates or judges the LLM judge?

Andrei: I’ve thought about this philosophically. There are two answers. First: a human has to put their judgment into the grading rules in the first place — that’s your “judge of the judge,” similar to how real-world judges are governed by law or a codex someone wrote. Second: you can use a stronger model to review a weaker judge’s scoring, the way a higher court can review a lower court’s rulings. But then you hit the same problem one level up — who judges the human? That’s where we still take our stand. The day LLMs take over that ultimate decision-making, we’re obsolete anyway.

John: After going through building your own AI SRE, what would you not build again?

Andrei: I’ve only built one — Klaudia. Knowing what I know now, I wouldn’t have moved from our own agent harness to a third-party library (Agno). We do enough non-standard things — parallel requests, mid-session context injection — that migrating to a generic library meant either giving up some of those tricks or hacking the library to support them anyway. Our development velocity took a hit because we weren’t confident enough to keep investing in our own harness. That said — we’re not in a rush, so we have time to build our own eventually. The idea that a library would save us time didn’t really pan out.

Audience question: What language is the AI SRE platform written in?

Andrei: The agent harness itself is in Python — most of Klaudia is Python, which is also the default choice for most coding agents today. But the surrounding platform involves a good amount of Go — our authentication and RBAC layer, the privilege-scoping I described earlier, and parts of the tool-calling and MCP layer are in Go, built for efficiency and scalability reasons that Python doesn’t handle as well.

Nikki: Thank you so much to Andrei, and to everyone who joined. We’ll follow up with everything, and this recording will be available for you to watch again. If you have more questions, send them our way. Have a great day.