Memory vs. History vs. Retrieval vs. Cache: A Clear Mental Model for AI Apps

Stop treating every piece of context the same. Here is how to separate turn history, RAG snippets, persistent user memory, and prompt caching.

Abstract editorial still life of four forms: a flowing time ribbon, a dense well, scattered retrieved sheets, and a crystal cache

When an AI app forgets a user preference or hallucinates a policy document, the root cause is usually architectural: developers mix up four distinct layers of context. In the updated Claude API course, we break these down into four separate jobs.

1. Conversation History (Working Memory)

History is the ordered list of user, assistant, and tool_result turns in the current session. It is ephemeral and bounded by the context window. When you compact older turns to save space, details are permanently lost.

2. Retrieval / RAG (External Reference Library)

Retrieval answers: What do our documents say about X right now? It indexes a shared corpus and injects cited passages (path + line numbers) at query time. It is not per-user state.

3. Persistent App Memory (Scoped User Facts)

Persistent memory survives process restarts and new conversations. Your application—not the model—owns the storage layer:

  • Per-user isolation: Store files under a scoped root like data/users/<id>/memories/ and reject .., absolute paths, and symlink escapes.
  • Expiry & deletion: Support TTL timestamps so temporary context ages out automatically.
  • Secret filtering: Block API keys and credentials from ever being written to memory files.

4. Prompt Caching (Transport Optimization)

Prompt caching doesn't change what the model knows—it lowers latency and cost when you resend the same static prefix (system instructions, tool schemas, or a large reference file) across multiple calls.

More stories