Our Prefix Cache Died at Token 1,067

A 47,000-token agent prompt with a carefully designed stable-first layout, getting 49.8% cache reuse. The culprit was upstream of the system prompt entirely.

Summary

Our agent framework sends ~47,000-token prompts and was getting 49.8% prefix cache reuse, with p90 time-to-first-token at 135 seconds. Four spans in the prompt mutate between turns. Three of them are in the volatile tail where they belong and do bounded damage. The fourth sits at token 1,067 — inside the tool definitions array, which the chat template renders before the system message — and it invalidates everything. About 27× of TTFT was sitting behind one of those four.

How we found it

Not by inference. We decoded token IDs straight out of the inference server's session bank — the actual stored prompts from real turns — and diffed two consecutive turns token by token.

That's worth recommending as a technique. Reasoning about which parts of your prompt vary is guesswork, and it's guesswork about a system with several layers of template rendering between your code and the tokens. Reading the tokens the server actually cached tells you the answer in one pass, with byte offsets.

Four spans differ between turn A and turn B.

The four, in the order that matters

1. Token ~1,067 — the tool catalog. The earliest, so it dominates everything.

A: {"name": "tool_search", "description": "Search 19 additional tools loaded on demand...
B: {"name": "web_extract", "description": "Extract content from web page URLs...

2. Tokens ~23,402 and ~24,597 — the skill catalog. Entries appear and disappear between turns as different capabilities load.

3. Token ~34,393 — a memory character counter.

A: MEMORY (your personal notes) [99% — 2,180/2,200 chars]
B: MEMORY (your personal notes) [98% — 2,161/2,200 chars]

A live utilization percentage rendered into a prompt header. It changes whenever the notes change, by a character.

4. Token ~35,406 — the date.

A: Conversation started: Wednesday, August 26, 2026 (EDT, UTC-04:00)
B: Conversation started: Tuesday, August 25, 2026 ...

The earliest volatile token sets the ceiling. Prefix caching is exact-match from position zero. Everything after the first difference is invalidated, so fixing spans 2, 3, and 4 while leaving span 1 in place buys exactly nothing.

The design was already correct

Here's the part that makes this interesting rather than just a bug report.

The framework's system prompt is deliberately built for cache stability. Its own source documentation says so:

The agent's system prompt is built once per session and reused across all turns — only context compression triggers a rebuild. This keeps the upstream prefix cache warm.

Three tiers, joined in order: stable (identity, tool guidance), context (system message, context files), volatile (skills index, memory snapshot, timestamp and session line).

The volatile tier is already last. Someone thought about this and got it right.

So culprits 2, 3, and 4 — skills, memory counter, date — sit in the volatile tail by design. Their damage is bounded to the tail and largely legitimate.

The killer isn't in the system prompt at all.

Token ~1,067 is inside the OpenAI-format tools array, which the chat template renders before the system message. The observed offsets confirm the ordering exactly: tools at ~1,067, then skills at ~23,402, then memory at ~34,393, then date at ~35,406.

And the tools array mutates mid-session, because the framework loads tools on demand. A tool gets deferred until needed; when it's needed, the array changes; the next turn's prompt differs at token 1,067; 100% of the prefix is invalidated — upstream of the carefully-designed stable tier, which never gets a chance to help.

The generalizable lesson: your prompt does not start where your system prompt starts. Tool definitions, and anything else the chat template emits ahead of the system message, are part of the cached prefix. A framework can implement stable-first layout perfectly within its own prompt and still lose the entire cache to a JSON array it doesn't think of as prompt content.

If you're auditing prefix stability, audit the rendered prompt, not the prompt you wrote.

The fix, ranked

Fix the tools array first. It's ~95% of the problem. Three viable options:

• Load the full tool catalog up front instead of on demand.
• Only ever append newly-loaded tools to the end of the array — never insert or reorder.
• Freeze the deferred-tool set for the session once it has been expanded.

The other three are already in the volatile tail and are second-order. Worth doing eventually, worthless on their own.

Expected: cache reuse 49.8% → ~99%, p90 TTFT 135 s → ~5 s.

No runtime can rescue this

Worth stating plainly, because it's the reason this matters more than most tuning work.

vLLM, SGLang, llama.cpp, and MLX-based servers all cap at 0–0.04% reuse when the prompt head changes. There is no cache implementation, no eviction policy, and no amount of memory that recovers a prefix whose first thousand tokens differ.

Prefix stability is the only lever. It's a property of the client, not the server.

Which makes it, for an agent workload, a bigger lever than anything available in the serving layer or the kernels. We'd spent weeks on engine evaluation, quantization, and scheduler tuning — all of it chasing single-digit or low-double-digit percentages. This one is 27×, and it's a change to how a JSON array is assembled.

An ordering trap in on-demand loading

The specific pattern here is worth naming because on-demand tool loading is becoming standard in agent frameworks, and it's a direct conflict with prefix caching.

The feature exists to keep the prompt small: don't send 40 tool definitions when the model needs 5. The cost is that the set of definitions becomes a function of conversation history, and therefore changes between turns, at the very front of the prompt.

Both goals are legitimate. They're just in tension, and the tension is invisible until you measure cache reuse. If you're building this, append-only is the reconciliation: expanding the tool set costs you the tail, not the head.

Key takeaways

1. Decode the tokens your server actually cached and diff consecutive turns. Reasoning about prompt volatility is guesswork; the byte offsets are free.
2. The earliest volatile token sets the ceiling. All fixes before it are worthless; the fix at it makes the others matter.
3. Your cached prefix starts before your system prompt. Tool definitions render first and are part of it.
4. On-demand tool loading conflicts directly with prefix caching. Append-only is the reconciliation.
5. No serving runtime can recover a changed prompt head. Every engine caps near zero. This is a client-side problem exclusively.
6. A correct design in one component can be defeated by a layer above it. The stable-first prompt architecture was right and still lost, because something upstream got prepended.