In the world of cloud-native infrastructure, complexity is the silent killer of innovation. For Cisco Outshift, the company’s incubation engine, managing a sprawling environment of AWS EKS clusters and edge-based MicroK8s workloads created a classic bottleneck: the Platform Engineering team was drowning in toil. Facing SRE burnout and the limits of human scaling, Cisco embarked on an ambitious journey to evolve its internal operations from standard DevOps to Agentic AI. The result is CAIPE (Community AI Platform Engineering) - an open-source, multi-agent system that leverages Komodor’s Klaudia to reduce MTTR by up to 80%. In this post, we break down the technical architecture behind Cisco’s solution, how they automate agent generation using OpenAPI specs, and the rigorous evaluation strategies required to put AI SREs into production. The Challenge: "Bottleneck by Design" Modern platform engineering is often a bottleneck by design. At Cisco, the infrastructure stack includes: Compute: AWS (EKS) and Edge (MicroK8s). Control Plane: Argo CD, Backstage, GitHub Actions. Observability: Splunk and Komodor. Hasith Kalpage, Director of Platform Engineering at Cisco, noted that scaling this stack with humans resulted in frustration and slow releases. The goal was to shift from reactive support to creative innovation. The team envisioned a system akin to J.A.R.V.I.S from Iron Man, a system where platform engineering is handled by intelligent, collaborating agents. The Architecture: CAIPE and Multi-Agent Orchestration Cisco built CAIPE as a multi-agent system that abstracts complexity through standardized protocols. The architecture relies on three core pillars: LangGraph & Deep Agents: Cisco migrated from simple chains to "Deep Agents" capable of long-horizon planning. A supervisor agent delegates tasks to sub-agents (e.g., a PagerDuty agent, an Argo CD agent, or a Komodor agent). MCP (Model Context Protocol): To standardize tool calling, CAIPE utilizes MCP. This allows the system to transform API endpoints into executable tools that LLMs can reason with. A2A (Agent-to-Agent): This communication layer allows distributed agents to collaborate. For example, a "Developer Agent" in VS Code can talk to a "Cluster Agent" running securely behind a firewall. The Role of Komodor and Klaudia AI Cisco integrated Komodor’s Klaudia, an autonomous AI SRE, as a specialized sub-agent. When a deployment fails, the CAIPE supervisor detects the issue via Argo CD and triggers the Komodor agent to perform a Root Cause Analysis (RCA). "It’s typically finding that needle in the haystack... these agentic systems are really good at finding that information quickly." — Hasith Kalpage Unlike a standard chatbot, Klaudia is designed to perform deep data synthesis across the Kubernetes stack. Needle in the Haystack: Hasith noted that the primary challenge in outages is finding the specific root cause among millions of events. Klaudia reduces this investigation time from hours to seconds.Context Awareness: Klaudia doesn't just look at logs; she correlates changes, configuration drift, and health signals across hybrid environments (e.g, AWS and Edge) to provide a complete RCA. Why Komodor? The Reliability Factor Arthur Drozdov, Agentic AI Engineer at Cisco, emphasized that in the world of MCP, reliability is scarce. "I can go out and find a hundred MCP servers that say they do something, but when you actually try to use them, it doesn’t work. Whereas with Komodor, it works reliably." - Arthur Drozdov This API reliability is critical. If the underlying tool fails or returns hallucinated data, the entire agent chain breaks. Komodor’s deterministic API responses allow the CAIPE agents to function with high confidence. The Workflow: Debugging in Production How does this look in a live scenario? In the webinar demo, the team showcased a "Debug the crashing pods" workflow: Trigger: A developer asks the CAIPE bot (via Slack, Backstage, or VS Code): "Show my apps and debug any failures". Orchestration: The Supervisor Agent analyzes the request and first calls the Argo CD Agent to identify the app status. Handoff: Upon detecting a CrashLoopBackOff in the dev environment, the supervisor delegates control to the Komodor Agent. Analysis: The Komodor Agent queries Klaudia. Klaudia analyzes the specific namespace, correlates recent deployment changes with the crash, and returns a synthesized Root Cause Analysis. Result: The user receives a plain-text summary of why the pod crashed and how to fix it, without ever opening a dashboard. Technical Implementation: Automating Agent Generation One of the most significant engineering breakthroughs in CAIPE is the ability to generate agents automatically from documentation. Instead of manually coding tool definitions for every API endpoint, Cisco utilizes OpenAPI specs. Arthur detailed the workflow using Komodor’s API as the blueprint: Ingest OpenAPI Spec: The team takes Komodor’s public OpenAPI specification. Code Generation: Using a custom tool (openapi-mcp-codegen), they parse the spec to generate a fully functional MCP Server. Agent Creation: The tool automatically creates the agent bindings, allowing the LLM to understand how to query the Komodor API (e.g., "Get cluster health" or "Show events for pod X"). This approach allows the AI to reason with the full breadth of the tool's capabilities without manual wrapper coding. Secure Low Latency Messaging (SLIM) To ensure these agents can talk securely across environments (e.g., from a laptop to a production cluster), Cisco implemented SLIM. This protocol provides end-to-end encryption for individual conversations, offering a higher security posture than standard HTTPS for sensitive agentic payloads. Production Readiness: Trust, RAG, and Evals A major barrier to AI adoption in SRE is trust. To prevent hallucinations ("garbage in, garbage out"), Cisco employs a rigorous reliability stack: RAG & GraphRAG: The agents don't just guess; they pull context from internal documentation and playbooks. GraphRAG is used to correlate fragmented data sources, building relationships between disparate infrastructure components. Deterministic Task Configs: For critical workflows, Cisco encodes "task configs" - sets of steps the LLM is guaranteed to follow, ensuring the agent adheres to standard operating procedures. Golden Datasets & LLM-as-a-Judge: Reliability is tested via continuous evaluation using Langfuse. The team builds a "golden dataset" of queries (e.g., "Debug the crashing pod in dev") and uses an LLM to judge the agent's response against expected outcomes. "Production reliability comes from curated knowledge + deterministic task configs + continuous evaluation." - Arthur Drozdov The Impact: 80% Reduction in MTTR The implementation of CAIPE and Klaudia delivered tangible operational improvements: Speed: Query responses dropped from hours (waiting for a human) to seconds. Toil Reduction: Tasks like provisioning LLM keys, setting up repos, or granting access were reduced from days to minutes. Recovery: The mean-time-to-recover (MTTR) saw an 80% reduction, as agents could instantly detect issues, visualize them in Komodor, and suggest fixes. The Future: Autonomous Self-Healing The vision for Cisco and Komodor goes beyond chatbots. The industry is moving toward bidirectional agent collaboration, where agents don't just answer questions but proactively reach out to one another to resolve issues. Imagine a future where a monitoring agent detects a drift, contacts the Komodor agent to investigate, and a remediation agent applies a fix autonomously. As Hasith notes, moving from "assistant" to "autonomous" is where the 100x productivity gains lie. Ready to start your Agentic AI SRE journey? Explore the open-source CAIPE project on GitHub. Deep dive into the technical implementation of Klaudia at Cisco. Learn more about Cisco's platform engineering transformation by watching our recent joint webinar. See Klaudia in action and learn how Komodor provides the visualization and troubleshooting intelligence that powers the future of reliability.
