Google dropped Gemini with a 1M token context window. Anthropic is pushing 200K. The AI community celebrated: "Finally, agents can remember everything!"
Except they can't. And throwing more tokens at the problem is like giving someone a bigger desk instead of a filing cabinet.
The illusion of infinite memory
A 1M token context window means your agent can hold roughly 750,000 words in a single conversation. That's about 10 novels. Surely enough for an AI assistant to remember your name, your preferences, and that you hate when it uses the word "delve."
But context windows aren't memory. They're short-term attention. Every time you start a new session, the window empties. Your agent wakes up with total amnesia, again.
Sessions reset
Most agent interactions are session-based. User opens chat, talks, closes chat. Next time? Fresh context. That 1M token window was useful for exactly one conversation.
Real memory persists across sessions. It's there tomorrow, next week, next month. Context windows don't do this.
Cost scales linearly (and brutally)
Stuffing a million tokens into every API call isn't free. At current pricing, a single 1M-token prompt costs roughly $10-15 with GPT-4 class models. Per request. Your agent checks the weather? $10. It looks up your name? $10.
Memory retrieval via semantic search costs a fraction of a cent. You query what you need, when you need it. You don't haul your entire life story into every conversation.
Retrieval degrades with size
Here's the part that really gets me: LLMs get worse at finding specific information as context grows. The "lost in the middle" problem is well-documented. Models struggle to retrieve facts buried in the middle of long contexts. A 1M token window isn't just expensive, it's less accurate for recall than targeted semantic search.
Stanford researchers found that models miss relevant information in the middle about 40% of the time in contexts beyond 100K tokens. Bigger window, worse memory. I find that genuinely ironic.
What actual memory looks like
Real agent memory needs persistence across sessions, selective recall of relevant information, and some way to weight which memories matter more than others. That's it.
This is what purpose-built memory services do. Store a memory with context and an importance score. Later, query semantically: "What does this user prefer?" and get back the 5 most relevant memories, not 750,000 words of everything that ever happened.
# Store a memory
curl -X POST https://api.memoclaw.com/store \
-d '{"text": "User prefers dark mode and hates the word delve", "importance": 0.9, "tags": ["preferences"]}'
# Recall later (different session, different day)
curl -X POST https://api.memoclaw.com/recall \
-d '{"query": "user UI preferences", "limit": 5}'
No million-token context needed. The agent gets exactly what it needs.
The MEMORY.md problem
Some developers hack around this with MEMORY.md files, a markdown file the agent reads at the start of every session. It works, barely. But it eats context tokens, doesn't scale, has no semantic search, and becomes a mess after a few weeks.
You keep piling papers on the desk until nothing is findable.
Context windows are still useful
I'm not anti-context-window. Larger windows are great for processing long documents in a single pass, maintaining coherence in extended conversations, and analyzing large codebases.
But they solve a different problem than memory. Confusing the two is like confusing RAM with a hard drive. Both store data. Only one survives a restart.
Where this leaves us
The fixation on context window size misses the point. Your agent doesn't need to hold everything in its head at once. It needs to remember selectively and recall intelligently.
1M tokens won't fix agent amnesia. Persistent, searchable memory will.
MemoClaw provides memory-as-a-service for AI agents. Store and recall memories with semantic search. No API keys, no subscriptions, free tier included.
Top comments (0)