AI agent memory architecture: why memory is a write-permission problem, not a storage problem

AI agent memory architecture is the design of what an agent may remember across runs, who is allowed to write it, with what evidence, for how long, and how you take it back. That framing is deliberate. Most memory frameworks describe storage: a vector store, a knowledge graph, a tiered cache modelled on an operating system. Storage is the easy part. The hard part is that in almost every demo the model itself decides what is worth remembering, summarises it, gives it authority over future decisions, and retrieves it later, with no reviewer at any step. Memory is a write-permission problem wearing a storage costume.

The stakes are documented. The MINJA attack showed that an ordinary user, with query access only and no access to the memory store, could plant records in an agent's memory that later steered other users' sessions, with an injection success rate above 95 percent and an attack success rate around 70 percent in the paper's settings. One injected memory becomes a standing capability. This article gives you the four things to keep separate (state, candidate pool, context, memory), the six memory classes and what each may authorise, a promotion lifecycle that turns "remember this" into a governed write, why compaction is a lossy build and caching is not remembering, how poisoning gets in, a fit table for Mem0, Cognee, Zep, Letta and home-grown, and how to evaluate the memory rather than the answer.

Here is the case we use in the harness playbook, because it fits in two sentences. A support agent resolves a duplicate-charge ticket; Leila in compliance approves a €240 refund as an exception, and the case closes. Six weeks later the agent, having "learned from experience", tells a different customer that refunds of €240 need no approval. Nothing malicious happened. The agent remembered an event and stored it as a rule. Remembering an event is not learning a rule, and a memory system that cannot tell the difference will eventually promote every exception into policy.

That is the whole argument. What follows is the machinery to stop it.

Keep four things separate

The easiest way to corrupt an agent is to collapse concepts that happen to be serialised together. The transcript is where all four of these get mixed, and the transcript is the wrong place for three of them.

Three rules fall out. State lives outside the transcript: if a worker crashes, the next one resumes from typed task and effect state, not from the last assistant message. Context is disposable: it exists for one inference and can be rebuilt, which is what makes policy changes, revocation and model swaps manageable. And memory is a claim. "The customer prefers email" is a preference claim. "Refunds under €200 need no approval" is a policy claim. They need different sources, owners, expiry and promotion paths. Storing both in the same vector collection does not make them the same kind of knowledge.

Six memory classes, and what each may authorise

Frameworks talk about short-term versus long-term, or episodic versus semantic, borrowed from cognitive science. Useful, but the question a production system needs answered is different: what may a retrieved memory of this kind do? Inform a decision, personalise within a scope, or authorise an action?

Read the right-hand columns and the €240 story explains itself. The refund was an episodic fact and the agent stored it as policy. The model may write working memory as freely as it likes, because working memory dies with the task. Everything below that row needs a writer who is not the model alone.

Memory promotion is a governed write

The most dangerous memory design in the field is one function: remember_if_useful(text). It lets the same model decide what is useful, summarise it, assign it authority over the future, and retrieve it later. One compromised interaction gets a long half-life. Replace the function with a lifecycle:

CANDIDATE -> QUARANTINED -> CORROBORATED -> ACTIVE
                 |               |              |
                 v               v              v
              REJECTED        REJECTED     SUPERSEDED -> EXPIRED

A candidate is what the model proposes at the end of a run. Quarantine means it exists but nothing may retrieve it as authority yet. Corroboration is the check that decides whether it earns activation: a second source, a verified effect, a user confirmation, or a human, depending on the class. Active memories carry an expiry and a supersession key so a newer fact can retire them cleanly. Every arrow is an attributable transition. The promotion record itself is small:

promotion:
  claim: "customer 4471 prefers email for billing notices"
  memory_class: preference
  purpose: personalise channel choice for this customer
  source_units: [msg://case-9182/turn-14]
  originating_principal: customer 4471 (authenticated)
  corroboration: customer_confirmed | second_session | human
  scope:  tenant acme; customer 4471; billing notices only
  valid_from: 2026-09-13
  expires: 2027-09-13         # or on customer edit
  supersedes: pref://4471/channel@v2
  may_inform: [channel_selection]
  may_authorize: []           # a preference never authorises an action
  owner: support_platform
  retraction: on customer request; on incident affecting case-9182

Two fields do most of the work. may_authorize is empty for every class except procedural and policy, and those two are never written from conversation at all. And source_units is what lets you quarantine the descendants when a source turns out to be poisoned. Rebuilding only the vector index leaves the summaries, plans and caches exactly as compromised as before.

The gate that enforces this is short enough to read in a minute:

FORBIDDEN_FROM_CONVERSATION = {"procedural", "policy"}

def promote(candidate, evidence, policy):
    cls = candidate.memory_class
    if cls in FORBIDDEN_FROM_CONVERSATION:
        return reject(candidate, "class not writable from a run")
    if candidate.may_authorize and cls != "procedural":
        return reject(candidate, "memory may inform, not authorize")
    if not candidate.source_units or not candidate.principal:
        return reject(candidate, "no provenance")
    if not policy.in_scope(candidate.scope, candidate.principal):
        return reject(candidate, "scope wider than the writer")
    if not evidence.corroborates(candidate, policy.required_for(cls)):
        return quarantine(candidate)        # exists, not retrievable
    record = activate(candidate, expires=policy.ttl(cls),
                      supersedes=find_conflict(candidate))
    audit(record)
    return record

Notice what the model does not get to do here. It proposes. It does not corroborate its own proposal, it does not set the scope, and it cannot write the two classes that carry authority. The enforcement sits in the harness, in a component whose decision does not depend on believing the model's summary, which is the same principle that governs tool permissions and action approval. It is the state plane of the harness, and it is why memory quarantine is one of the twelve properties we test a harness against.

Compaction is a lossy build, and caching is not remembering

Long-running agents outgrow one context window, and two mechanisms get mistaken for memory. The first is compaction. MemGPT showed the value of managing tiers of in-context and external memory, and provider APIs now expose compaction directly; OpenAI's documentation is candid that the compacted item is opaque, not something a human can read. Compaction solves continuation. It does not turn the compressed artifact into operational truth. A safe compaction boundary externalises before it compresses: the current task state and owner, accepted requirements and their versions, unresolved conflicts, proposals and approvals with their hashes, dispatched and unknown external effects, exact identifiers and amounts, remaining budgets. Then the summary may compress the discussion, the discarded hypotheses and the redundant tool output. Treat a context reset like a process restart: rebuild from durable state, validate the new package, and test whether the obligations from before the reset survived. If the summary is the only place an approval or an unresolved payment exists, the design failed before compaction began.

The second is caching. Prompt and prefix caches cut repeated computation. They do not create memory, and a cache hit does not make content current. A cache key should bind at least the tenant, the effective permission scope, the task class, the source and policy versions, and the compiler version. Do not put a user identifier in the key and call the cache isolated; the payload has to be partitioned and re-authorised at read time. Research presented at NDSS 2026 reconstructed other tenants' prompts through shared KV-cache side channels in evaluated serving frameworks. Cache optimisation belongs below the same permission boundary as retrieval, not above it.

How memory poisoning gets in

Persistent memory changes the attacker's horizon. They no longer need to control the victim's current request; they plant a record that will be retrieved later. MINJA (NeurIPS 2025) did this through the front door: an attacker interacts through the normal query interface, uses "indication prompts" to get the agent to generate bridging steps that link legitimate requests to the malicious content, then progressively shortens the prompts until the poisoned memory is retrieved by ordinary victim queries. Over 95 percent injection success and about 70 percent attack success in the evaluated systems, with no access to the store. AgentPoison (NeurIPS 2024) and PoisonedRAG (USENIX Security 2025) showed the complementary paths through retrieval corpora and knowledge bases.

A follow-up in January 2026 tested these attacks under more realistic conditions and found that pre-existing legitimate memories dilute them considerably; the idealised numbers are an upper bound, not a forecast. It also proposed defences built on trust scoring and temporal decay. Those help, and they are not the boundary. The boundary is the promotion gate above: a write policy at least as strict as the read policy, provenance on every record, instruction fields kept separate from factual claims, and one rule carved above the door. Do not let the same model generate a lesson, declare it safe, promote it, and retrieve it later as authority. Four jobs, four different writers. When an incident happens, quarantine everything descended from the poisoned source, which is what source_units is for.

Framework fit: who reviews the write?

The frameworks are good at storage and retrieval, and they differ mostly in the shape of the store. The column that matters for this article is the one none of them advertise.

This is not a criticism of the frameworks. Their job is to make memory work; the governance is the application's job, and the docs mostly say so. The trap is assuming that because a framework has "memory" in the name, the write side has been thought about. Pick the store for the shape of your data (a graph for relationships that change over time, vectors for recall, files for things humans should read), and put the gate in front of it regardless. Our own Cognee walkthrough and the Agno guide cover the storage side; this article is what sits in front of them.

Evaluate the memory, not the answer

LongMemEval, the benchmark most teams reach for, grades five things: extraction, reasoning across sessions, temporal reasoning, knowledge updates, and abstention. The lesson in that list is that memory quality includes updating and declining to answer, not just recalling an old statement. An enterprise evaluation has to add four more: permission (did a retrieved memory cross a tenant or scope?), provenance (can every active memory name its source?), deletion (how long after a retraction does the memory stop influencing decisions?), and poisoning resistance (run the MINJA pattern against your own agent and measure how many planted records reach activation).

Compare against strong baselines, in order: a full-history dump; a permission-filtered top-k retrieval with no memory at all; obligation-aware retrieval without memory; the full system with governed memory. Keep the negative results. When we built the first version of our own context-governance architecture, the recorded four-way experiment showed the trivial ACL-filtered retrieval baseline with zero measured leaks and the elaborate architecture with 307 leakage events over 200 queries. The right response was to kill the architecture that lost and keep the invariants that were worth keeping, and the same discipline applies to memory: if governed memory does not change a qualified outcome on a task class, that class does not need durable memory. The evaluation method, graders and promotion gates, is what the Context Architecture course is built around, with the governance-as-code version in our Context Kubernetes paper.

Looking at unusual stores? Memvid packs a vector index into a video file. Fun, fast, and it still needs a gate on the write side.

What I'd do

Separate the four objects first, on paper: where does state live, what may the model inspect, what does it receive for this decision, and what is allowed to persist. Then classify every memory your agent currently writes into the six classes and check the two right-hand columns; anything in the procedural or policy rows that was written from a conversation gets deleted today. Put the promotion gate in front of the store you already have, with quarantine as the default landing state and corroboration rules per class, and make the model's job proposing only. Externalise state before every compaction and treat a context reset as a restart. Run MINJA against your own agent before someone else does, and add a "retraction latency" number to your dashboard. Write the memory policy down as a versioned document the team runs on, the same idea as the AI Operating System we teach operators, because the prompt was never the bottleneck and neither is the vector database. And if you would rather build the governed version with us, that is Context Architecture.

FAQ

What is AI agent memory architecture?

The design of what an agent may remember across runs, who may write it, with what evidence, for how long, and how it is retracted. It separates system state, the candidate pool, model context and memory, classifies memories by what they may authorise, and treats every durable write as a governed promotion with provenance, scope and expiry.

What is the difference between short-term and long-term memory in agents?

Short-term or working memory lives inside one task and dies with it; the model may write it freely. Long-term memory persists across runs and includes episodic (what happened), preference, semantic (facts about the world), procedural (how to do things) and policy classes, each with different sources, owners and authority. The useful distinction is not duration but what a retrieved memory may do.

What is memory poisoning?

An attack that plants records in an agent's memory so they steer future sessions. MINJA showed it can be done through the normal query interface with no access to the store, with over 95 percent injection success in the evaluated systems. The defence is a promotion gate: no unverified model output writes directly to durable memory, every record carries provenance, and descendants of a poisoned source can be quarantined.

Cognee vs Mem0: which should I use?

Cognee builds a knowledge graph plus vectors from documents and conversations and suits relationship-heavy recall. Mem0 extracts facts per conversation and lets an LLM decide add, update or delete, and suits user-preference memory. Neither reviews writes; both need a promotion gate in front of them. Choose by the shape of your data, then govern the write side yourself.

Is RAG the same as agent memory?

No. Retrieval finds candidates in a corpus you curated; memory is what the agent itself accumulates from its runs. Retrieval quality is about relevance and freshness. Memory quality is about who wrote the record, whether it was corroborated, what it may authorise, and whether it can be retracted. A retriever score says nothing about any of that.

How do you roll back a bad memory?

Every active memory carries its source units and a supersession key. Retract by marking the record superseded or expired, then quarantine every record and derived artifact that lists the same source, including summaries, plans and caches. Rebuilding only the vector index is not a rollback. Measure retraction latency: how long after the retraction the memory stops influencing decisions.

Leila's €240 exception is still in the case history, where it belongs, as a fact about one case on one day. It never became a rule, because the system it runs in cannot turn an event into policy without a policy owner writing the policy. That is the entire design, and it fits in a sentence.

Caching is not remembering, and remembering is not learning. Learning is a write someone approved.

Charafeddine MouzouniSeptember 13, 2026