Moving the agent off your machine and pointing it at real clusters — read-only, in shadow mode — then climbing a trust ladder toward carefully scoped action. This is the second article in a three-part series on taking an AI SRE agent from a weekend experiment to enterprise production. Part 1 built a local agent on a throwaway cluster: read-only, propose-only, refined against a small eval set, with portable skills and no production write access. Part 3 will cover enterprise hardening — airgapped deployment, least-privilege access across the org, governance, and scaled autonomy. This piece is the leap between them: getting the agent running in the cloud, against real infrastructure, without inheriting a blast radius you can't control. Two things change the moment you leave the laptop, and both raise the stakes. The agent now runs on its own — deployed off your laptop, event-driven, and stateful, rather than waiting for you to paste an alert into a terminal. And it has to face real, messy, adversarial production signals instead of the failures you injected and already understood. The gap between "it nailed the OOM I deliberately caused" and "it's right about a live incident at 2:17am" is exactly the gap this article is about closing. So the governing idea of Part 2 is restraint with a plan. You do not flip a switch from laptop experiment to trusted operator. You climb a trust ladder one rung at a time, and the eval loop you built in Part 1 is what issues the permits. By the end you'll have a cloud-deployed agent investigating real incidents in shadow mode — fully traced, continuously evaluated against real ground truth — with a clear, evidence-gated path toward scoped remediation on non-critical systems. Humans stay firmly in the loop for anything that matters. Graduate from the IDE to a runtime Claude Code was the right harness for experimentation because it let you build behavior without first building infrastructure. But you can't keep a terminal open forever, and production work is event-driven and long-running. The first move in Part 2 is to re-house the agent in a runtime that runs it as a service: persistent, triggered by events, with its own compute. You have good options, and the choice mostly follows your ecosystem: Claude Agent SDK — the programmatic version of Claude Code. You keep the same mental model, the same skills, and the same MCP connections, and deploy the agent as a long-running service. The path of least surprise if Part 1 felt natural. Deep Agents (LangChain, on LangGraph) — the open-source recreation of Claude Code's architecture, model-agnostic, with LangGraph's persistence, streaming, and checkpointing available out of the box. Strands Agents (AWS) — model-driven, with first-class deployment to Lambda, Fargate, EKS, and Bedrock AgentCore, plus a session manager for state. Microsoft Agent Framework — if you live in Azure or .NET and want durable workflows with an LTS commitment. Open SRE — a batteries-included AI SRE agent on LangGraph you can fork instead of building, with integrations for the alerting and observability sources you already run. The reassuring part: this migration is not a rewrite. Skills follow the Agent Skills open standard, so the ones you wrote in Part 1 move over unchanged. Your prompts and CLAUDE.md context carry forward. And most importantly, your eval set comes with you — you're re-housing behavior you already validated, not starting from zero. If you want a reference for what a well-built deployed agent looks like, LangChain's Open SWE is a coding agent rather than an SRE one, but its deployment patterns — isolated sandboxes, OAuth, a credential proxy, parallel runs — transfer directly to what you're building. In fact, it maps cleanly onto the remediation half of an SRE agent: where Open SRE handles triage and RCA, a coding agent like Open SWE can open pull requests to fix the issues it finds, so it converts naturally into the acting side of an AI SRE. Give it durable state Local runs were effectively stateless: one alert in, one answer out. A real incident is long-running and multi-step — the agent forms a hypothesis, gathers more evidence, revises, and may run for many minutes across several tool calls. That needs durable state. LangGraph gives you checkpointing and resumable runs; Strands offers a session manager that pulls state from a remote datastore. Wire it so the agent can persist progress and resume rather than losing its place when a process restarts. You don't have to build all of this from scratch: existing tools like Jira, GitHub Issues, or ClickUp are an easy place to start for tracking issues and progress, and they double as a memory the agent can search later — so the next time a similar incident fires, it can surface the old issue and reuse the fix it already applied. Monitors and triggers for this phase In Part 1, triggers were local toys: a cron job, a Flask listener on your laptop, a GitHub webhook tunneled through smee. This phase replaces them with real, deployed monitors — but with one piece of discipline that's specific to where you are on the trust ladder. You do not wire the agent to every alert. Because the agent is running in shadow mode and earning trust incrementally, the set of monitors that triggers it is itself a scoped, rung-gated decision: start with a narrow band of well-understood alerts on non-critical services, and widen the aperture only as the agent proves itself on what it already sees. Pointing it at the full firehose on day one guarantees noise and buries the signal you're trying to evaluate. Concretely, the loop looks like this: wire the agent to one narrow webhook — both to control cost and to build trust deliberately — ideally a rare but high-signal alert like a service being down that's well worth testing against; let it run triage, RCA, and a suggested recommendation; and track the quality of those outputs over time, using GitHub to tie each change back to the improvement it produced. The monitors worth wiring as triggers are the ones your team already trusts. On the reactive side: Prometheus Alertmanager, Datadog Monitors, Grafana Alerting, PagerDuty or Opsgenie events, CloudWatch Alarms, and Sentry issues. On the proactive side, two trigger types are worth setting up now, both in observe-only mode — scheduled health sweeps via cron (the agent does a periodic read-only pass and reports anything that looks off), and post-deploy verification fired from your CD system or a GitHub/GitLab webhook (the agent investigates whether a rollout degraded anything). Open SRE already integrates most of these sources; if you're building it yourself, the frameworks give you the webhook and event plumbing. Whatever you wire up, put cost controls around it from day one: cap spend per session and enforce a global budget so a runaway agent can't quietly rack up token costs — an increasingly real failure mode as teams lean harder on coding and SRE agents. Routing is where most of the work is, and it pays to do the filtering before the agent ever wakes up. Use Alertmanager's grouping, inhibition, and silences so the agent receives deduplicated, grouped signals rather than raw noise — an agent that turns a hundred-alert storm into a hundred investigations is worse than no agent at all. Scope the route narrowly and keep humans on the parallel path, since shadow mode means the on-call still gets paged exactly as before: # alertmanager.yml — route only a narrow, well-understood set of alerts to the # agent at first. Widen this matcher as the agent climbs the trust ladder. route: receiver: oncall-default routes: - matchers: - severity = warning - service =~ "cart|recommendation" # non-critical services only - alertname =~ "PodCrashLooping|HighMemoryUsage" receiver: sre-agent # shadow-mode investigation endpoint continue: true # humans still get paged in parallel receivers: - name: oncall-default pagerduty_configs: [ ... ] - name: sre-agent webhook_configs: - url: https://agent.internal.example.com/alert send_resolved: true The ingress itself is no longer a laptop process — it's a deployed, durable endpoint, and that changes its requirements. It has to verify webhook signatures (PagerDuty, GitHub, and Datadog all sign their payloads), so a forged request can't trigger an investigation; this is part of the same secrets-and-auth discipline from the shadow-mode section. It has to be idempotent, because the same alert refires and you don't want duplicate investigations racing each other — dedupe on a stable alert fingerprint. And rather than processing inline, prefer to enqueue: a queue decouples bursty alerts from agent runs, lets you rate-limit to control cost, and gives you a clean place to isolate each incident in its own run. Treat every new monitor you add to the route as a small expansion of the agent's scope — gated, like every other rung, by how well it has handled the alerts it already receives. Point at real clusters — read-only, shadow first This is the discipline that makes the rest safe. Connect the agent to real observability and cluster APIs through MCP, with scoped, read-only credentials, and run it in shadow mode: the agent investigates real, live incidents and produces a hypothesis and a proposed fix, but it takes no action, and the on-call human resolves the incident exactly as they would today. The agent's output is posted alongside the incident, clearly labeled as observe-only. Shadow mode is the unlock, because it gives you something the Part 1 sandbox never could: ground truth on real traffic. Your injected failures had known causes you wrote yourself; real incidents have causes and resolutions discovered by your team under pressure. When you compare the agent's hypothesis to what actually fixed the problem, you learn whether it's right about the world you actually operate — not the toy world you built. Run it as a progression, not a leap. Start against a staging or non-production cluster, move to production read-only shadow, and only then begin thinking about writes. Stay in production shadow long enough to accumulate a real track record — weeks and many incidents, not a handful of lucky runs. And handle credentials the way Open SWE does: keep secrets out of the agent's execution sandbox, prefer short-lived scoped tokens, and route access through a proxy where you can. The telemetry the agent reads is attacker-influenced content, so a compromised or prompt-injected run must not be able to escalate or exfiltrate. # shadow.py — run the agent on a REAL alert, observe-only, then score it later. # No remediation tools are loaded. The agent investigates; humans resolve. async def on_real_alert(alert): hypothesis = await sre_agent.investigate( alert, tools=READ_ONLY_TOOLS, # logs, metrics, deploy history — no writes permission_mode="propose_only", ) await store.save({ "alert_id": alert.id, "alert": alert, "hypothesis": hypothesis, # cause, blast radius, proposed fix "trace_url": hypothesis.trace, # OTel / LangSmith trace for replay "action_taken": None, # shadow mode: the agent never acts }) await oncall_channel.post(hypothesis, tag="shadow — for comparison only") # When a human resolves the incident, attach ground truth and feed the # pair into the eval dataset. This is how shadow mode trains your evals. async def on_incident_resolved(alert_id, actual_cause, actual_fix): record = await store.get(alert_id) await eval_dataset.add( input=record["alert"], agent_hypothesis=record["hypothesis"], ground_truth={"cause": actual_cause, "fix": actual_fix}, ) Trace the agent itself On your laptop you could watch the agent reason in real time. In the cloud it runs unattended, and you cannot operate a non-deterministic system you can't see into. Instrument it with OpenTelemetry, and use LangSmith, Langfuse, or Komodor Agentic Operations for agent-specific trace views and run replay. Capture every tool call and its result, every handoff, the decision points, token usage and cost, and end-to-end latency. This is Part 1's "if you didn't capture it, it didn't happen" scaled to production. When the agent posts a wrong hypothesis on a real incident — and it will — you need to pull up that exact run and see why: which log it over-weighted, which tool returned garbage, where the reasoning turned. Those traces are also the raw material for the next section, because the run you replay to debug is the same run you promote into your eval set. Keep the eval loop running — now fed by reality In Part 1 your golden set was injected failures with causes you already knew. Shadow mode replaces that with something far richer: real incidents with real resolutions. Promote those traces into your eval dataset (the snippet above shows the hook), and your evaluation stops being a synthetic check and becomes a measure of real-world performance. Make evaluation continuous. Every change to a prompt, a skill, or a tool re-runs the suite, and a regression blocks the change. Keep tracking the two numbers from Part 1 — did it name the right cause, and would you have approved the fix it proposed — and add the ones that only matter once you're against real traffic: the false-positive rate (is the agent adding noise?), time-to-hypothesis, and the quality of the evidence it cites. These numbers are not a report card you read after the fact. They are the permits for the trust ladder: you expand the agent's reach when the metrics clear a bar you set in advance, never on a hunch after a good week. The trust ladder: graduated autonomy Here is the heart of Part 2. Earning the right to act is a climb, and each rung is gated by evidence — eval numbers, a clean observability story, and a rollback plan — before you take the next step. flowchart TB R1[Rung 1: Read-only shadow on non-prod] R2[Rung 2: Read-only shadow on production] R3[Rung 3: Propose-only on production] R4[Rung 4: Approved remediation, non-critical only] NEXT[Part 3: enterprise production grade] R1 -->|right-cause rate + evidence quality| R2 R2 -->|weeks of real incidents, low false-positive rate| R3 R3 -->|high would-approve rate + rollback + traces| R4 R4 -.->|governance, RBAC, airgapping, scaled autonomy| NEXT Read the rungs concretely. Rung 1 is the agent investigating staging incidents and taking no action — your shakeout against a safe target. Rung 2 moves the same observe-only behavior to production: it sees real incidents, humans resolve everything, and you build a track record. Rung 3 is propose-only on production — the agent posts a specific proposed fix to the on-call channel, and a human decides whether to run it; the agent still never acts, but its proposals are now part of the workflow. Rung 4 is the first time the agent executes anything: approved remediation, restricted to low-blast-radius actions on non-critical services, and only after explicit human approval — the propose-then-approve-then-act pattern from Part 1, now against real (but carefully chosen) systems. And here is where Part 2 deliberately stops. Fully autonomous remediation across production, and the governance, access control, and isolation required to support it responsibly, is Part 3. If you find yourself wanting the agent to fix production by itself after a strong run at Rung 4, that instinct is the subject of the next article — not this one. Pitfalls and challenges to overcome The failure modes here are higher-consequence versions of Part 1's, plus a few that only appear once real systems are in scope: The blast-radius jump. The single biggest risk is treating the cloud deployment like the laptop. The signals are real, the consequences are real, and the casual "let it try" habit that was harmless on a kind cluster is not harmless here. Secrets in the sandbox. Don't hand a long-lived production credential to the agent's execution environment. Scope tokens tightly, keep them short-lived, and proxy access — the Open SWE pattern exists precisely because the sandbox is the thing most likely to be compromised. Adversarial telemetry. Prompt injection stops being theoretical once the agent reads real logs and tickets that outside parties can influence. Read-only access, scoped tokens, and human-gated writes remain your defense; the trust ladder is built around it. The agent as a new source of noise. A chatty agent that posts confident, wrong hypotheses erodes on-call trust faster than almost anything. Tune for precision, measure the false-positive rate explicitly, and let the agent stay quiet when it isn't sure. Over-trusting a short shadow run. A week of good RCAs is not a mandate for write access. The whole point of setting eval bars in advance is to hold them when a good run tempts you to skip a rung. State and concurrency bugs. Multiple incidents at once mean shared state, races, and crossed wires. Isolate runs — a sandbox per incident, Open SWE style — so two investigations can't contaminate each other. Cost surprises. An unattended agent reacting to every alert can burn tokens quietly. Budget it, rate-limit it, and watch the per-run cost your tracing already captures. The meta-pitfall, again, is skipping rungs. Every incident the agent "would have caught" creates pressure to hand it the keys. The ladder exists for exactly that high-pressure moment. What's next You now have a cloud-deployed AI SRE agent that is event-driven and stateful, investigating real incidents in shadow mode, fully traced and continuously evaluated against real ground truth, and climbing an evidence-gated trust ladder toward carefully scoped remediation on non-critical systems — with humans in control of anything consequential. That is the bridge between experimentation and production. In Part 3, we make it enterprise-grade: airgapped and self-hosted deployment for regulated environments, least-privilege access and SSO across teams, governance and audit you can put in front of a compliance review, continuous-learning loops turned into durable infrastructure, and the scale to run across many teams and clusters. That's the hardening that turns a trusted tool into infrastructure the whole organization can rely on.
