The first stage of AI SRE maturity is a laptop, a throwaway cluster, and zero production access. Here's how to set it up, and what to watch for. AI SRE (Site Reliability Engineering) agents are AI-powered systems that automate the most time-consuming parts of incident response: triaging alerts, correlating logs and metrics, generating root-cause hypotheses, and proposing remediation steps. The problem they solve is a scaling one — modern infrastructure generates far more signals than any on-call engineer can meaningfully process at 2.17am. Rather than replacing the human, an AI SRE agent acts as an always-on first responder that shows up with evidence, not noise, so the engineer can make a faster, better-informed decision. This is the first article in a three-part series on taking an AI SRE agent from a weekend experiment to something an enterprise can run in production. Part 2 leaves the laptop, moving to the cloud and pointing the agent at real clusters. Part 3 is the enterprise hardening: airgapped deployment, least-privilege access, continuous learning, and governance. This piece is about where everyone should actually start, and where too many teams don't: the local experimentation stage. It's worth being blunt about why that matters. Gartner's late-2025 survey of infrastructure and operations leaders found only 28% of AI use cases in I&O fully meet their ROI expectations, with failures clustering in auto-remediation and agent-led workflows. A big reason is that teams wire an agent into production before they understand how it behaves, they skip the cheap stage where mistakes cost nothing, and go straight to the one where mistakes page someone. The local stage is where you build intuition and trust in a system you can break and recreate at will. By the end of this article, you'll have a local AI SRE agent that investigates simulated incidents in a throwaway environment, packaged into reusable skills, triggered by local events, and refined against a small evaluation set, without a single byte of production access. Why start on your laptop The case for the local sandbox is the same case you'd make for a staging environment, only more so. Iteration is instant, no deploy, no waiting. The blast radius is zero; the worst the agent can do is mangle a cluster you were going to delete anyway. And crucially, this is where you learn the failure modes that actually bite in production: agents that hallucinate confident-but-wrong root causes, agents that get steered by a malicious string in a log line, agents that reach for kubectl delete when they should have proposed a rollback. You want to meet all of those for the first time on your laptop, not anywhere near production. So the local stage runs on two rules: read-only first, and propose, don't execute. Build those habits now, and they carry into every later stage. Build with Claude Code locally You could start by assembling an agent loop,tool-calling, and a context manager from scratch. Don't, not yet. Claude Code already gives you the agent loop, tool use, MCP integration, skills, subagents, and hooks. At this stage, your job isn't to build agent infrastructure; it's to build agent behavior. Claude Code lets you skip straight to that. Install it from npm: npm install -g @anthropic-ai/claude-code Then give it the context an SRE would need. Spin up a dedicated subagent repo — a new repository that contains only the agent's CLAUDE.md, its skills, and its hooks. Use the CLAUDE.md to describe your stack, your conventions, what "healthy" looks like, and where the dashboards live. Connect read-only data through MCP servers: point Claude Code at a local Grafana or Prometheus, or a read-only Kubernetes MCP server, so it can pull real evidence instead of guessing. Keep every connection read-only to start. Now run the cheapest feedback loop in existence: paste an alert into an interactive session and ask Claude to investigate. Watch how it reasons, which tools it reaches for, and where it goes off the rails. That observation is the whole point of this phase. It helps to know Claude Code's extension layers, because they do different jobs and you'll use all of them: CLAUDE.md is always-on project context; MCP connects external tools and data; skills are on-demand expertise loaded only when relevant; subagents isolate a noisy sub-task in its own context; and hooks move enforcement into the harness so a rule holds deterministically regardless of what the model decides. Most of the work below lives in skills and hooks. Experiment with skills to automate routine tasks An SRE's day is full of repeatable procedures: triage a failing service, investigate a latency spike, or hunt down a resource leak, diff the last few deploys, check resource saturation, and draft an RCA from the evidence. Each of those is a candidate for a skill. A skill is just a directory containing a SKILL.md file: YAML frontmatter with a name and a description, followed by Markdown instructions, plus optional scripts/, references/, and assets/ folders. The architecture is built on progressive disclosure. Claude loads only the name and description of every skill at startup, pulls the full body into context when a task triggers it, and runs bundled scripts without loading them into context at all. That means deterministic, repeatable steps (the exact kubectl or PromQL invocations) belong in scripts/, where they're cheap and consistent, while the judgment stays in the prose. Two things matter most when authoring. First, the description is the trigger; it's not documentation, it's what Claude matches against to decide whether to load the skill, so make it specific and a little pushy ("Use this whenever a workload is crash-looping, a service is returning errors, or resource saturation is detected..."); vague descriptions are the single most common reason a skill never fires. Second, scope the tools: the allowed-tools frontmatter field (supported in the Claude Code CLI) lets a triage skill read and run diagnostics without ever being able to mutate anything. Here's a minimal triage skill: --- name: triage-pod-crash description: > Investigate a crash-looping or restarting pod. Use this whenever a pod is in CrashLoopBackOff,OOMKilled,restarting repeatedly,or failing readiness , even if the user just pastes an alert and doesn't name the skill. allowed-tools: Read,Bash(kubectl get:*),Bash(kubectl describe:*),Bash(kubectl logs:*) --- # Triage a crashing pod 1. Identify the workload and namespace from the alert. 2. Run `scripts/collect.sh ` to gather pod status,recent events,container last-state,and the last 200 log lines. 3. Correlate the failure with the most recent rollout (image tag,config change). 4. Output an evidence-backed hypothesis: probable cause,blast radius,severity (SEV1–SEV4). Cite the specific log line or event for every claim. 5. Do NOT propose or take any remediation. Investigation only. Keep personal experiments in ~/.claude/skills/ (available across all your projects); once a skill earns its keep, promote it to .claude/skills/ in the repo so the team shares it via git. A good rule of thumb is to keep each SKILL.md body under ~500 lines and push anything heavier into references/. And because skills are portable across Claude.ai, Claude Code, the Agent SDK, and the Developer Platform, the ones you write here travel with you into the later stages of the series. Once a skill is stable enough to trust, share it with your team. The fastest way to build org-wide confidence in AI SRE is to let a small inner circle run the same skills against their own sandboxes, compare notes, and surface edge cases you haven’t hit yet. Skills are portable — commit them to the shared .claude/skills/ directory in the repo and anyone pulling the latest will have them on their next session. Start with the people who are already comfortable with the agent and let them become advocates before you roll out broadly. Create your own triggers and monitors So far, you’ve been invoking Claude manually. In Part 2, we’ll cover how to wire up event-driven triggers and monitors — letting the agent respond automatically to real alerts from your observability stack. Run it locally in a non-mission-critical environment To investigate incidents, you need incidents, so manufacture them safely. Spin up a throwaway local cluster and give it something to break: kind create cluster --name sre-sandbox # Deploy a demo app with realistic failure surfaces,e.g. Online Boutique: kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/main/release/kubernetes-manifests.yaml # Now break it on purpose: kubectl set image deployment/checkoutservice server=busybox:bad-tag # bad image kubectl set resources deployment/cartservice --limits=memory=8Mi # force OOM kubectl scale deployment/recommendationservice --replicas=0 # take it down Because you injected the failure, you know the ground truth, which makes this the perfect place to grade the agent. Point your read-only triage agent at the sandbox, let it investigate, and check whether its hypothesis matches what you actually did. Hold the line on remediation: even here, keep the agent in propose-only mode behind a human approval gate. There's no production to protect on a kind cluster, but the discipline of "the agent proposes, a human approves, then a separate step acts" is exactly the muscle you want trained before Part 2. And to be explicit about the boundary of this article, a throwaway local cluster you can kind delete and recreate is the playground. Pointing the agent at real, shared, or production clusters is where Part 2 begins. Evaluate it At this stage, “tuning” means running evals and iterating on the levers that actually move quality: the system prompt and CLAUDE.md, the skills (their descriptions and their steps), tool scoping, and the context you feed in. You are shaping behavior, not model weights. Literal fine-tuning (SFT on your own data) is expensive, freezes the model’s behavior, and is premature when you don’t yet have a crisp definition of “good” — evals are how you build that definition.. Make your evals evidence-driven rather than vibes-driven. Take the incidents you injected into the sandbox, each with a known cause, and turn them into a small golden set. Replay them after every change and score the results. Promptfoo is an open-source evaluator that runs assertions and LLM-as-judge checks and slots into CI; Langfuse (self-hostable) and LangSmith capture traces and let you build eval datasets from real runs. Track two numbers above all: did the agent name the right cause, and would you have approved the fix it proposed? Then iterate against those numbers. When a skill under-triggers, tighten its description. When the agent's reasoning wanders, move the deterministic part into a script or enforce it with a hook. When it invents a plausible-sounding cause, constrain it to cite the specific evidence for every claim, and re-run the golden set to confirm you fixed it without breaking something else. This eval loop is the single most valuable artifact you build in Part 1, because it's the thing that tells you, later, whether the agent is actually ready to leave the laptop. The local experimentation loop ties together like this: flowchart LR TRIG[Local trigger: cron / file watch / Alertmanager webhook] --> CC subgraph CC[Claude Code -p: read-only tools + hooks guardrail] INV[Investigate and hypothesize] end KIND[(kind / minikube cluster + demo app,broken on purpose)] OBS[(Local Prometheus / Grafana / Loki)] CC -. read-only .-> KIND CC -. read-only .-> OBS CC --> RCA[RCA / hypothesis output] RCA --> HUMAN{You review: right cause? approve fix?} HUMAN --> GOLD[(Golden incident set)] GOLD -. replay and score .-> EVAL[promptfoo / Langfuse] EVAL -. refine skills + prompts .-> CC Pitfalls and challenges to overcome A few hard-won words of caution, most of which are easier to learn now than later: Logs are untrusted input. The logs, traces, and tickets your agent reads are attacker-influenced content, and a crafted string in a log line can carry a prompt injection. Treat tool output as untrusted, and let that be another reason the agent stays read-only with human-gated writes, even locally. Confident hallucinations. Agents produce authoritative-sounding root causes that are simply wrong. Forcing evidence citations helps, but the real defense is the golden set: it's what catches confident-but-wrong before it becomes a habit you trust. The demo always works. One impressive run is not a track record. The strongest discipline you can adopt is to graduate the agent on eval numbers, not on the memory of a demo that dazzled. Don't grant write access early. The temptation to let the agent "just fix it" is strong and almost always premature. Propose-don't-execute is the rule until you've earned the right to relax it, which is not in Part 1. Capture everything. Non-deterministic systems are only debuggable if you logged the run. For your own incident reviews, if you didn't capture the structured output and the trace, it didn't happen. Curate tools, mind context, and cost. Resist tool sprawl; a focused toolset beats a large one, and progressive disclosure keeps the context window sane. Headless loops can also quietly rack up tokens, so watch the per-run cost that --output-format json reports. Skills run with your privileges. A skill, especially a third-party one, can invoke shell commands and read sensitive files. Audit any skill before you install it, exactly as you would a dependency. Keys Leakage An agent without proper guardrails is a security liability. Because it has read access to your environment — config files, environment variables, mounted secrets — a rogue or prompt-injected agent can exfiltrate API keys, service tokens, and credentials by writing them to logs, embedding them in RCA output, or sending them to an external endpoint. The risks are real: a crafted string in a log line can redirect the agent’s next action, and an over-permissioned skill can read files it has no business touching. Scope every skill’s allowed-tools to the minimum it needs, never mount secrets into the agent’s working directory, treat all tool output as untrusted, and keep the agent in propose-only mode until you’ve audited its behavior against a golden eval set. What's next You now have a local AI SRE agent that investigates simulated incidents in a throwaway cluster, packaged as reusable skills, triggered by local events, refined against an eval set, and constrained by a propose-don't-execute habit, all with zero production access. That's stage one of AI SRE maturity, and it's the foundation on which everything else is built. In Part 2, we leave the laptop: moving the agent to the cloud, giving it durable state- and event-driven triggers from real alerting, and pointing it, carefully, read-only first, at real clusters. That's where the distributed-systems problems begin, and where the eval loop you built here starts earning its keep.
