Codifying Tribal Knowledge – How to Build Long Term Memory for AI Agents

In this webinar, Nir Adler from Komodor explains how to build long-term memory for AI agents, specifically focusing on Site Reliability Engineering (SRE). He details the architecture needed to extract, segment, and retrieve contextual memories so that AI agents do not start every debugging session with “amnesia.” By implementing continuous maintenance loops like consolidation and decay, organizations can drastically speed up AI-driven troubleshooting and improve resolution accuracy.

TL:DR; Breakdown

• Speaker: Nir Adler, AI Innovation Engineer at Komodor, specializing in developing AI agent initiatives and open-source tools.

• Focus: Overcoming AI “amnesia” by designing a robust, long-term memory system that retains tribal knowledge for SRE agents.

• Core Concepts: Effective agent memory relies on proper indexing, context-specific retrieval, and continuous maintenance (consolidating duplicates and decaying outdated records).

• Includes: An architectural overview of how memories are extracted post-run, categorized by domain, and automatically retrieved to accelerate future debugging.

• Wrap-up: Implementing an optimized agent memory reduces average resolution times by over 100 seconds while boosting overall response quality.

Key Takeaways

• Stateless AI agents waste time by re-deriving the same conclusions for recurring issues, highlighting the critical need for a dedicated memory layer.

• A good agent memory distills raw run transcripts into short, reusable insights rather than just saving massive, unindexed log files.

• Proper memory segmentation—such as labeling by cluster, environment, or service—ensures agents only retrieve contextually relevant data.

• Maintaining an active memory storage requires regular “garbage collection” to consolidate duplicate entries, reinforce helpful tips, and delete unused information.

Webinar Transcript 

Please note that the following text may have slight differences or mistranscription from the audio recording.

Host: Welcome everyone to today’s webinar from Komodor: Codifying Tribal Knowledge: How to Build Long-Term Memory for AI Agents.

A quick housekeeping note: this webinar is being recorded. You can use the Q&A section to ask questions throughout the presentation, and we will address all of them at the end. The webinar will run for approximately 45 minutes. We will also send out the full recording and slide deck within a couple of days.

Our speaker today is Nir Adler, AI Innovation Engineer on Komodor’s CTO team. Nir works on every AI agent initiative across our engineering team. I will hand it over to Nir to introduce himself and dive into the topic.

The Problem: AI Root-Cause Tools Wake Up with Amnesia

Nir Adler: Thanks everyone. I work on the CTO team at Komodor, where I focus on open source development, constantly building new tools and researching AI agent systems. Today, we are going to talk about agent memory: why we need it, how it works, how to implement it correctly for your specific use cases, and how it leads to better operational outcomes.

Every AI agent starts by default with a completely clean context window—it essentially wakes up with amnesia. As human engineers, when we approach a problem, we bring our past experiences and context. We recall how a previous issue was resolved and apply those learned patterns to the task at hand. Default AI agents lack this ability.

Some agent harnesses attempt to implement local, project-based memory tied strictly to a single code repository or directory. While that works when staying inside one repository, complex production systems span multiple repositories, platforms, and microservices. When an incident spans across these boundaries, repository-isolated agents cannot access cross-cutting insights, causing them to get stuck or re-investigate known issues from scratch.

What Makes a Memory Good & Industry Best Practices

Nir Adler: Managing agent memory comes down to two primary requirements: saving and indexing memory correctly, and retrieving it at the exact right moment.

If an agent retrieves irrelevant memories during an active investigation, it gets confused—much like a human distracted by irrelevant facts. Supplying bad or unrelated context causes the agent to debug down the wrong path, increasing resolution time rather than shortening it.

We drew inspiration from several industry research papers and frameworks:

• Google’s ReasoningBank: Techniques for distilling concise memory notes out of long execution sessions.

• AWS Tiered Memory: Strategies for separating short-term context from long-term storage.

• Letta / MemGPT: Implementations of agents managing their own memory states.

• Neo4j Agent Memory: Structuring memories and entity relationships through graph representations.

The key lesson is that you must tailor your memory strategy to your agent’s domain. For instance, a Security Operations Center (SOC) agent might require entity-relationship graphs, whereas an SRE agent needs quick, keyword-and-fuzzy searchable diagnostic playbooks.

Inside Agent Memory: The Extraction & Retrieval Flow

Nir Adler: When an agent runs, it executes numerous tool calls, inspects logs, and runs diagnostics over a long, context-heavy session.

Once that run completes, an asynchronous pipeline triggers:

1. Extraction: A separate LLM evaluates the completed session transcript.

2. Distillation: Rather than saving the full raw transcript, it distills the lesson into a short, reusable rule (e.g., “When Pod X fails to schedule, check if Node Y failed to join the cluster due to missing configuration options”).

3. Indexing: The distilled note is tagged with relevant metadata, such as cluster, namespace, and service name.

On subsequent runs, when a similar incident triggers an investigation, the active agent is presented with memory titles and summaries in its opening system prompt or through explicit tool calls. The agent then decides whether to fetch the full text of a specific memory note to guide its debugging path.

What a Memory Carries

Nir Adler: Good memory notes do not store static, fragile commands like “always restart payment-pod-123.” Static rules quickly become outdated and misleading.

Instead, a high-value memory carries procedural, relational knowledge: “When Service A experiences timeouts, check dependent Service B’s health logs first.” This procedural context remains accurate over time, preventing the agent from pursuing dead-end paths.

Segmenting Operational Knowledge

Nir Adler: To prevent memory pollution, memories must be categorized and segmented properly. For SRE workflows, operational knowledge generally maps across 8 distinct categories:

1. Failure Correlations: Components and symptoms that consistently fail together.

2. Resolution Playbooks: Steps that successfully resolved prior incidents.

3. Blast Radius: Downstream impact when a specific service degrades.

4. Capacity Patterns: Resource limits, replica counts, and auto-scaling behaviors.

5. Structure & Topology: Infrastructure dependencies and network traffic paths.

6. Temporal Patterns: Recurring issues tied to deployments, cron jobs, or peak traffic.

7. Tribal Knowledge: Team ownership, escalation paths, and unwritten operational rules.

8. Operational Constraints: Hard limits, baseline configs, and known environmental quirks.

Segmenting memories by these categories and tagging them with environment metadata ensures the agent retrieves only the facts pertinent to its current target environment.

Memory Lifecycle & Continuous Maintenance

Nir Adler: Unchecked memory accumulation leads to bloated database stores, wasted prompt tokens, and context window pollution. Memory systems require an active maintenance lifecycle:

• Consolidation: Merging duplicate or highly similar memory notes into unified canonical playbooks.

• Reinforcement: Increasing the rank and confidence score of memories that repeatedly assist agents in successful investigations.

• Decay & Archival: Automatically decaying and archiving unused, unconfirmed, or outdated memories so they fade out of the active retrieval set.

We execute daily scheduled passes to clean up duplicate entries, rewrite outdated notes, and enforce memory decay.

Performance Impact & Benchmarks

Nir Adler: In head-to-head evaluations across 5,164 production comparisons, equipping SRE agents with structured long-term memory yielded measurable performance gains:

• Speed: Saved an average of ~110 seconds per Root Cause Analysis (RCA) investigation (242.7s with memory vs. 352.5s without memory).

• Quality: Delivered a +4.3% increase in final investigation accuracy and response quality.

Questions & Answers

Host: Thank you, Nir. Let’s open the floor to questions. To kick things off: from your experience building these systems, what is a major pitfall or dead end that teams should avoid when implementing agent memory?

Nir Adler: The biggest pitfall is relying solely on a raw vector database to store full session transcripts. Dumping entire execution transcripts into embeddings creates massive, noisy memories. When an agent retrieves a full past transcript, it tends to blindly replay the exact same long, turn-by-turn debugging steps instead of taking the shortest path. You must distill the session down into a short, concise playbook note before storing it.

Host: We have a question from the audience: How do you prevent sensitive information or secrets from being stored in memory?

Nir Adler: We enforce a three-tier defense strategy:

1. Regex Scrubbing: Automated pattern matching strips standard API keys, tokens, and credentials.

2. Prompt Instructions: The distillation LLM is explicitly instructed to exclude secrets and sensitive payload data when writing memory notes.

3. Maintenance Review: The daily background maintenance pass reviews stored notes as a secondary guardrail to catch and purge any residual sensitive data.

Host: Another question from Hugo: Can an agent generalize from one failure and apply that lesson to similar situations, or only to the exact same use case? How do you prevent outdated failure memories from degrading decisions?

Nir Adler: Memories are specifically prompted to be generalized playbooks—toolboxes rather than hardcoded static answers. Instead of saving “restart Pod X,” the memory records the diagnostic methodology to identify root dependencies.

To prevent outdated memories from degrading performance, our background maintenance process tracks feedback. If a retrieved memory fails to help resolve an incident or leads to a dead end, its usefulness score drops, causing it to decay and eventually be archived.

Host: Question from Vjoja: Are users able to manually manage memories, such as updating or deleting them?

Nir Adler: Yes. While automated background processes handle most maintenance, we provide user interfaces that allow team members to view, edit, add, or delete memories directly. Engineers often know their environment’s nuances best, so empowering them to manually edit or insert tribal knowledge rules accelerates agent learning.

Host: That wraps up our Q&A session. Thank you Nir, and thank you to everyone for attending today’s webinar!

Frequently Asked Questions (FAQs)

Why is storing full execution transcripts in a vector database problematic for AI agents?

Storing raw, unindexed transcripts pollutes the agent’s context window with noisy log data and irrelevant tool outputs. This causes the agent to re-execute lengthy, step-by-step investigation paths rather than directly leveraging the distilled diagnostic solution.

How does an SRE AI agent know which memory to pull during an incident?

Memories are tagged with domain metadata such as cluster names, environments, service dependencies, and operational categories. When an incident occurs, the system runs a multi-stage retrieval pipeline (combining exact keyword matching, fuzzy search, and semantic similarity) to present only the most relevant memory summaries in the agent’s opening prompt.

What mechanism prevents incorrect or outdated memories from persisting indefinitely?

Agent memory uses an automated maintenance lifecycle featuring memory decay and consolidation. If a stored memory goes unused over time or fails to lead to successful incident resolutions, its confidence score decays until it is automatically archived from the active search pool.