BLOG

Your LLM Gateway Is Breaking Prompt Caching

One missing HTTP header cut my coding agent's cache hit rate to 55%. Why LLM gateways break prompt caching, and why prefix hashing can't save you.

If you live in Claude Code or Codex CLI, prompt caching is a solved problem you never have to think about.

And you really don't want to think about it, because coding agents are abusive without it. A 25-step bug hunt that grows from 20,000 to 100,000 tokens sends 1.5 million input tokens over the run. 1.4 million of those tokens are the exact same conversation history resent on every step. When prompt caching works, the GPU keeps the key-value (KV) attention state from earlier turns in VRAM and only runs prefill on the new tool output at the tail. At a 90% cache discount on Anthropic and OpenAI, you pay for 240,000 tokens instead of 1.5 million.

When you talk straight to Anthropic or OpenAI from their own CLI, the harness and the API handle all of that behind your back.

Then you try a different harness like Pi or OpenCode, or you wire up your own LangGraph agent, and you run it at work behind an internal LLM gateway. And suddenly you are reading raw HTTP headers at 11 PM trying to figure out why a GPU cluster forgot a conversation it had nine seconds ago.

A custom gateway URL breaks the handshake

Inside most companies, you don't point your agent straight at api.anthropic.com or inference.baseten.co. You point it at an internal gateway (something like LiteLLM, Portkey, Envoy, or an in-house proxy) that gives everyone a single /v1 URL and routes across OpenAI, Anthropic, Google, and open-weight models on providers like Baseten.

I'll use a LiteLLM-backed gateway as the example here. When I pulled a month of my Pi (v1.0.4+) session logs (~4,800 Turn-2+ requests, 584M prompt tokens) through that setup, OpenAI and Google models were hitting cache on more than 93% of warm tokens. Open models hosted on Baseten (zai-org/GLM-5.3 and zai-org/GLM-5.3-Flash) were sitting between 31% and 55%.

50% is a terrible number to debug. If your cache hit rate is 0%, you know it's broken. At 50%, you see cache hits in every session and figure the misses are just context growth or 5-minute TTLs expiring while you read code.

So I dumped a single 12-turn GLM-5.3 session turn by turn.

Why did Turn 8 remember Turn 2, but forget Turns 3 through 7?

Every turn in this session fired within 9 to 34 seconds of the previous one. Nobody went to get coffee. Nothing expired. And cacheRead did this:

t=0  | gap= 0.0s | tot_prompt=26,989 | cacheRead=     0
t=1  | gap=31.9s | tot_prompt=38,369 | cacheRead=26,902  <-- Worker A (has t=0)
t=2  | gap=16.9s | tot_prompt=44,056 | cacheRead=26,902  <-- Worker B (only has t=0, missed t=1)
t=3  | gap=26.2s | tot_prompt=51,503 | cacheRead=38,370  <-- Worker A (matches t=1's 38,369)
...
t=7  | gap=13.3s | tot_prompt=76,213 | cacheRead=67,004  <-- Worker A (matches t=6)
t=8  | gap= 9.1s | tot_prompt=78,529 | cacheRead=44,057  <-- Worker B (matches t=2's 44,056 from 6 turns ago!)
t=9  | gap=33.9s | tot_prompt=85,944 | cacheRead=76,214  <-- Worker A (matches t=7's 76,213)

Look at t=8. Nine seconds earlier, t=7 read 67,004 tokens from cache. Then t=8 comes along with a 78,529-token prompt and reads... 44,057.

Where did 44,057 come from? Look six lines up at t=2: tot_prompt=44,056.

t=8 had landed on a GPU worker that hadn't seen my session since t=2. Its VRAM still held my conversation prefix up to Turn 2, and it had to re-prefill the 34,472 tokens from Turns 3 through 8 from scratch. Nine seconds later, t=9 bounced back to the first worker and picked right up at 76,214 (matching t=7).

Dumping the HTTP response headers from the gateway caught the workers red-handed:

x-litellm-model-api-base: https://inference.baseten.co/v1
llm_provider-x-baseten-dyn-prefill-worker-id: 5073481647418772908
llm_provider-x-baseten-dyn-prefill-dp-rank: 0

Baseten returns the upstream prefill worker ID (llm_provider-x-baseten-dyn-prefill-worker-id) and data-parallel rank (dp-rank) on every response. In a 6-turn test with default settings, my requests pinballed across four different prefill workers.

Diagram comparing multi-turn agent requests bouncing across three unpinned prefill workers without x-session-affinity versus staying pinned to Worker 1 when x-session-affinity is sent

The annoying part is that both Pi and Baseten had already fixed this for direct connections:

  • If Pi sees baseten.co in your provider's baseUrl, it automatically turns on sendSessionAffinityHeaders: true and stamps x-session-affinity: <sessionId> onto every request.
  • If you connect Claude Code, Codex, or OpenCode straight to Baseten, Baseten recognizes their native session IDs and pins the replica for you.

Putting a gateway in the middle blinded both sides. Pi saw https://llm-proxy.internal/v1 instead of inference.baseten.co, assumed it was talking to a generic OpenAI endpoint, and left sendSessionAffinityHeaders off. LiteLLM got a request with no session header and forwarded it upstream as a stateless call.

Flipping "sendSessionAffinityHeaders": true on in ~/.pi/agent/models.json fixed it in one line:

{
  "providers": {
    "internal-gateway": {
      "baseUrl": "https://llm-proxy.internal/v1",
      "api": "openai-completions",
      "compat": {
        "sendSessionAffinityHeaders": true
      }
    }
  }
}

In the 6-turn header test, every turn stayed glued to a single prefill worker. On an 8-turn GLM-5.3-Flash benchmark, Turn-2+ cache hits jumped from 55.5% to 92.9% (+37.4 percentage points), and uncached prompt tokens dropped by 6.3x (from 97,958 down to 15,617).

Chart showing turn-by-turn cache hit rate rising from 45-83% (55.5% overall) to 91-94% (92.9% overall, +37.4 percentage points) across turns 2 through 8 after enabling x-session-affinity

That could have been the end of the post. Add a header, get 93% cache hits, go home.

Except that response header bothered me.

Wait. Doesn't Baseten already run a KV-aware router?

Look at the header name again:

llm_provider-x-baseten-dyn-prefill-worker-id: 5073481647418772908

That dyn in dyn-prefill-worker-id is NVIDIA Dynamo. Baseten has written whole blog posts about running Dynamo's KV-aware router. Their BIS-LLM engine docs explain how their stateful router tokenizes each incoming prompt, checks a live index of every worker's KV cache blocks, and sends the request to the worker with the most cached tokens (balanced against queue depth).

If Baseten already has a router that knows which GPU holds which KV blocks, why do their Model API docs still tell you to send x-session-affinity? Why can't the provider just figure out that Turn 2 belongs with Turn 1 by looking at the prompt?

Let's try to design content-based routing ourselves and watch where it breaks.

Attempt 1: Hash the first 256 tokens (the OpenAI way)

OpenAI doesn't ask you for a session header on basic calls. Their router hashes the opening prefix of the prompt, typically the first 256 tokens, scoped to your organization.

When it's just you on your personal API key, that works great. Turn 1, Turn 2, and Turn 20 all start with the same system prompt, so they hash to the same machine.

Now put 500 engineers behind a company gateway that authenticates to the provider with one shared enterprise API key. At 9 AM, 50 engineers fire up Pi or Claude Code. Every single one of those sessions starts with the exact same 256-token system prompt.

If your router only hashes the first 256 tokens:

  1. All 50 engineers hash to the exact same machine.
  2. That machine saturates. On OpenAI, once a prefix crosses roughly 15 requests per minute, the router spills excess traffic onto secondary machines.
  3. Because the router is only looking at the first 256 tokens (which are identical for all 50 engineers), Engineer A's Turn 2 lands on Machine 2 while Engineer B's Turn 2 lands on Machine 1. Everyone's conversation tail misses.

That spillover problem is why OpenAI added prompt_cache_key to their API. It mixes a caller-supplied session ID into the prefix hash so 50 sessions sharing a system prompt spread across the fleet while each individual session stays sticky.

Attempt 2: Hash the system prompt plus the first user message (messages[:2])

Okay, hashing messages[0] collides because everyone shares the same system prompt. What if the gateway hashes messages[0] plus messages[1] (the user's opening question in the thread)?

OpenRouter actually uses this as a fallback when you don't send an explicit session_id or prompt_cache_key. Two people rarely type the exact same first question, and in a normal chat thread messages[:2] stays fixed from Turn 1 to Turn 50.

In agent workloads, messages[:2] still blows up in three places:

  • Sliding-window history truncation: A lot of chat backends and LangGraph agents cap context by slicing the last N messages (messages[-10:] or trim_messages(..., strategy="last")). The moment your session hits Turn 11, Turn 1 gets dropped. Now messages[1] is different on every single request. Not only does that scramble a messages[:2] routing hash, shifting the array left by one turn invalidates the entire KV prefix of the remaining history on every turn (dropping history cache hits to 0% unless you evict in chunks of 5 or 10 turns at a time).
  • Mutated system prompts: Agents love stuffing live state into messages[0]: current workflow step, progress percentage, timestamps, or freshly retrieved RAG chunks. When messages[0] changes per turn, the hash changes per turn. (Fun side effect on OpenAI models: OpenAI serializes tools before messages[0]. When messages[0] mutates, your tool schemas still hit cache at ~20%. If your dashboard only checks cached_tokens > 0, everything looks fine while 100% of your system prompt and history miss.)
  • Slash commands and subagents: Automated PR-review bots and /review slash commands start every run with identical messages[:2] text, putting you right back in the hot-spot bucket. Meanwhile, spawned subagents start with a fresh messages[:2] that doesn't match their parent.

Why Baseten, Fireworks, and Cloudflare split routing into two tiers

Once you see all the ways content hashing goes sideways, the design that Baseten, Fireworks AI, and Cloudflare Workers AI converged on makes a lot more sense. Cloudflare summed it up in one line: "While we have KV-aware routing internally, we also rely on clients sending the x-session-affinity in order to be explicit about prompt caching."

Putting Baseten's Model API docs next to their BIS-LLM docs reveals a two-tier setup:

Diagram of two-tier routing showing Tier 1 L7 Edge Router using x-session-affinity to pick a Replica unit, and Tier 2 Dynamo KV Router inside Replica 1 tokenizing the prompt to pick the warm GPU worker
  1. Tier 1 (Outer L7 load balancer across replicas): Popular models scale across multiple independent replication units. At the outer HTTP edge, you do not want your load balancer parsing and tokenizing a 100k-token JSON body before it can pick a backend cluster. Tier 1 hashes x-session-affinity in constant time and pins your session to Replica 1.
  2. Tier 2 (Stateful KV-aware router inside the replica): Once your request is inside Replica 1, that replica's Dynamo KV router tokenizes the prompt, checks its local worker pool's KV block index, and picks the warm GPU worker (spilling over to a sibling worker in the replica if the queue is full).

That two-tier split also explains why my unpinned sessions were getting 31% to 55% cache hits instead of 0%. Whenever two turns happened to land on the same replica by luck, the inner KV router found the warm worker. Whenever the outer L7 balancer sent a turn to a different replica, that replica had an isolated KV pool and missed.

Can the LiteLLM gateway just inject the header for us?

If the upstream provider wants x-session-affinity, you might wonder why the company LiteLLM proxy doesn't generate one on the fly.

I went through the BerriAI/litellm source code to see what stock LiteLLM actually supports via config.yaml today:

  • Forwarding an existing session ID (PR #41033): LiteLLM has a provider_affinity_header: x-session-affinity setting for model deployments. If the caller sends x-litellm-session-id or metadata.session_id, LiteLLM copies it into x-session-affinity. If the caller sends neither, LiteLLM sends no header at all (Issue #34766 is the open feature request to add OpenRouter's messages[:2] fallback).
  • Proxy-level deployment pinning: LiteLLM also has a router flag (optional_pre_call_checks: [session_affinity, deployment_affinity]) to pin a session to one deployment inside a multi-provider model group. Turning that on globally has some nasty sharp edges right now: if a pinned deployment dies, in-request retries keep hammering the dead deployment until cooldown triggers (Issue #47482), and if a pinned deployment is saturated at its concurrency limit while a sibling sits idle, the pin refuses to yield and returns 429 (Issue #45868).

Passing x-session-affinity from the client through to Baseten avoids those LiteLLM router bugs when a model maps to a single Baseten deployment, because Baseten's outer load balancer treats the header as a soft hint and routes around dead replicas automatically. The catch is that header pass-through won't help if your LiteLLM model group load-balances the same model slug across two completely different providers (say, splitting traffic between Baseten and Cognition). And since x-session-affinity only controls routing, pausing for ten minutes on a busy pool still gives the worker plenty of time to evict your KV blocks from VRAM.

One last trap: LangGraph's thread_id never leaves your database

Even after you know you need x-session-affinity, agent frameworks make it easy to think you're sending a session ID when you aren't.

In LangGraph, every multi-turn tutorial has you pass a thread_id in RunnableConfig:

agent.stream(
    {"messages": [HumanMessage(content=user_query)]},
    config={"configurable": {"thread_id": thread_id}},
)

You'd think that thread_id makes it onto the LLM request. It goes to LangGraph's local state checkpointer and stops there; ChatOpenAI sends no affinity header on the wire. Worse, if you try to bind the header with llm = ChatOpenAI(...).bind(extra_headers={"x-session-affinity": thread_id}) and hand llm to langchain.agents.create_agent(model=llm, tools=tools), create_agent unwraps the binding when it calls .bind_tools(tools) and drops your extra_headers on the floor.

In a 3-turn, 6-call create_agent tool loop on GLM-5.3-Flash (langchain v1.0, langchain-openai v0.3), default ChatOpenAI scattered the six calls across four Baseten workers for a 29.8% warm-call cache hit rate (25,856 / 86,747 tokens). Passing default_headers directly on the ChatOpenAI constructor (or passing extra_headers to llm_with_tools.ainvoke() inside a custom StateGraph node) pinned all six calls to one worker and brought the warm-call hit rate to 87.9% (76,288 / 86,772 tokens, hitting 99% of the prior turn's prefix on every step):

llm = ChatOpenAI(
    model="zai-org/GLM-5.3-Flash",
    base_url="https://llm-proxy.internal/v1",
    default_headers={"x-session-affinity": thread_id},
)
agent = create_agent(model=llm, tools=tools, system_prompt=STATIC_PROMPT, checkpointer=checkpointer)

You can't fix what you don't measure

And breathe.

What we walked through here is only the tip of the iceberg. There are so many quiet inefficiencies hiding in our stacks right now because the entire space is brand new. The agent frameworks are new, the gateways are new, and every week brings another provider, another wire protocol, and another layer of routing. Nobody can hold all of those interactions in their head at once.

If you take one thing from this post, go audit your own agentic setup. Don't just check cached_tokens == 0 and call it a day. The expensive misses are the sinister ones that sit at 40% or 55% because your tool schemas or system prompt cached while your conversation history or worker affinity silently broke.

If you run AI infrastructure inside a company, open your token monitoring dashboard (hopefully you have one) and build a dedicated view for Turn-2+ cache hit rates broken down by model, provider, and harness. Find the combinations that are lagging and point your coding agents at the logs and gateway code to figure out why (or just tell your coding agent to read this amazing post). A one-line config change can save millions of dollars in wasted prefill, but you can't fix what you don't measure.


References