A 31.6% Hit Rate Is Both a Win and a Loss

A cache miss costs more than not caching at all. Set a breakpoint and the missed input bills at the write rate rather than list. Which invites a threshold: “turn it on above X% hit rate.”

I derived that threshold. Then I shook the conditions one at a time and it kept moving. At the same 31.6% hit rate, turning caching on is a loss when the hits bunch into a session’s early turns and a win when they bunch into the late ones. Same hit rate.

That last line is what shakes the threshold. One miss is worth 9.5 times another depending on where in the session it happened, so a single number called “hit rate” cannot fix the cost.

There are three axes: where hits fall inside the session, the turn count, and the tenant structure. The first two move the threshold; the third makes it meaningless. Below I size each one, and at the end I give the way to decide without converting to a threshold at all.

Inputs and terms

Same inputs as part two, on Claude Opus 5.

Input	Value	Basis
Stable prefix S	10,000 tokens	browser toolset 6,610 + bash 325 (documented) + system prompt 3,065 (assumed)
Per-turn increment δ	3,000 tokens	assumed
Output per turn	400 tokens	assumed
Turn count N	20 (base case)	assumed

The two setups being compared are these. Prefix-only puts a breakpoint on the stable prefix alone and lets the history pass at list price. Rolling moves the breakpoint to the tail of the history on every request, reading everything up to the previous one from cache and writing only what is newly added. Every comparison in this post is between those two. The “breakpoints on both the preface and the history” that appears once below is a third setup, there only to show what a miss costs.

A turn is one API request (same as part two). Not one user message. As shown below, the turn count is one of the axes that moves the threshold, so putting a user-message count here throws it off by multiples.

Rates are input $5, 5-min write $6.25 (1.25x), 1-hour write $10 (2x), read $0.50 (0.1x), output $25 — all per 1M tokens. The write and read multiples, and the fact that a missed input bills at the write rate, are all documented. (Anthropic pricing docs, prompt caching docs, retrieved 2026-08-23)

400 output tokens per turn works out to $0.20 of output cost in a 20-turn session. Per-session amounts include that output cost; amounts given per request or per user (the stampede, profile and tenant tables) do not.

How far sharing reaches is not an assumption — it is documented. Caches are isolated per workspace. The Claude API, Claude Platform on AWS, and Microsoft Foundry isolate per workspace; Amazon Bedrock and Google Cloud isolate per organization. Different organizations never share caches even on identical prompts. (prompt caching docs, retrieved 2026-08-23)

That lands on multi-tenant architecture. Give each tenant its own workspace and preface-tape sharing across tenants disappears. Inside a tenant it still works, so the cost of writing the preface multiplies by tenant count rather than user count. If tenants are separated at the application layer inside one workspace, everything is shared and this post’s arithmetic holds as written. Which case you are in sets the size of axis 3, so establish it before calculating.

The preface tape is alive, so S is read every turn

Caching is an exact prefix match. So a byte-identical stable prefix is shared across users, while the history differs per user and splits the cache. One preface tape; as many body tapes as there are people.

The preface tape survives on a single request within five minutes. With U active users requesting at an average interval of T minutes, the aggregate rate is U/T requests per minute.

Active users	5-min average interval	18 min	60 min
10	2.00 /min	0.56 /min	0.17 /min
100	20.00	5.56	1.67
1,000	200.00	55.56	16.67

Above one request per five minutes — 0.2 per minute — it stays alive. Only ten users at hourly intervals fall below that. At production scale the preface tape is not the problem.

So every calculation below assumes the preface tape is already in cache when a session starts. S is read at $0.50 every turn and no session writes it. That is also the accounting the docs specify: everything up to where the cache hit reaches bills as a read, and only what comes after it bills as a write. With a hot preface that point is the end of S, so only the history is written. The case where this premise fails is handled separately in step 0 of the last section.

The problem is always the body tape. For it to be alive, that same user has to come back within the TTL. Other people’s traffic does not help.

A miss costs more than not caching

Put a breakpoint down and that span becomes cacheable. A hit reads at 0.1x, but a miss bills at the write rate, not list, because the work of putting it in the cache is attached.

20-turn session	Input rate	Cost	vs. no caching
No breakpoint	$5.00 (1.00x)	$4.05	—
Breakpoints on preface and history, all missing (5-min)	$6.25 (1.25x)	$5.01	+23.8%
Breakpoints on preface and history, all missing (1-hour)	$10.00 (2.00x)	$7.90	+95.1%

The bottom two rows have the preface tape erased every turn too. Under the previous section’s premise — the preface is alive — these figures do not arise.

Fix the conditions and a threshold appears

Set a base case: 20 turns, the preface tape alive, and every gap inside the session drawn from the same distribution. Sweep the hit rate p under those conditions and you get two straight lines.

Hit rate p	Rolling, 5-min TTL	Rolling, 1-hour TTL
0%	$3.86	$6.00
24.2%	$3.15	$4.82
50%	$2.39	$3.56
58.5%	$2.14	$3.15
75%	$1.65	$2.34
90%	$1.21	$1.61
100%	$0.91	$1.13

Prefix-only costs $3.15, so the 5-minute threshold is 24.2% and the 1-hour threshold is 58.5%. If gaps are exponential, hit rate = 1 − exp(−TTL / mean gap), which makes those in-session average gaps of 18.1 and 68.3 minutes.

Both thresholds come out of the same expression. Work the algebra through and S and δ cancel, leaving only this:

threshold = N(w − L) / ((N − 2)(w − r))
   w = write rate, L = list input rate, r = read rate

5-min   N=20, w=6.25 -> 25 / 103.5 = 24.2%
1-hour  N=20, w=10   -> 100 / 171  = 58.5%

Stable prefix of 10,000 or 30,000, per-turn increment of 500 or 10,000 — the threshold is the same. Only the three rates and the turn count enter.

That is the answer with the conditions fixed. Now shake them.

Axis 1 — hit position moves the threshold the most

The table above treats the hit probability as the same on every turn. Real workloads are not.

Under rolling, whether a request hits or misses, the cumulative prefix as of that request — that is, everything up to and including it — stays in the cache. So the next request hits if its gap from the previous one is inside the TTL, and what happened before that does not matter. That much holds even when the gaps are correlated.

The problem is that one miss is worth a different amount on each turn. A miss rewinds the body tape and re-records from wherever it stopped, and turn 3 against turn 20 is a factor of 9.5. So the same hit rate produces different costs depending on whether the hits bunched early or late.

6 of 19 hits (31.6%)	20-turn cost	vs. prefix-only
Hits bunched in early turns (2-7)	$3.60	+14.4%
Uniform — what the threshold table assumes	$2.93	−7.0%
Hits bunched in late turns (15-20)	$2.26	−28.3%

The table says a 7.0% win where the reality can be a 14.4% loss. A 21-point spread.

The band where the decision actually flips is a hit rate of 26.3% to 47.4%. In that band the table says win and the early-bunched reality is a loss. The 52.6% / 26.3% / 15.8% figures come from integer hit counts out of 19, which is why their precision differs from the continuous 24.2% — the point where the table starts claiming a win is 5 hits out of 19, or 26.3%.

This is the most dangerous place to be using a converted threshold.

Axis 2 — fewer turns raise the threshold

The turn count enters the threshold expression directly. A shorter history means less to put in the cache and a larger share carried by the preface tape, which makes it harder for rolling to win.

Turns N	5-min threshold	5-min tolerated gap	1-hour threshold	1-hour tolerated gap	5-min ↔ 1-hour crossing
5	36.2%	11.1 min	87.7%	28.6 min	6.00 min
6	32.6%	12.7 min	78.9%	38.5 min	4.46 min
8	29.0%	14.6 min	70.2%	49.6 min	3.28 min
10	27.2%	15.8 min	65.8%	55.9 min	2.76 min
20	24.2%	18.1 min	58.5%	68.3 min	1.91 min
50	22.6%	19.5 min	54.8%	75.5 min	1.39 min
100	22.2%	19.9 min	53.7%	77.9 min	1.16 min

All five columns move with turn count. Apply the 20-turn values to a 5-turn service and you take the 1-hour threshold to be 58.5% when it is really 87.7%, so across that band you keep rolling on and lose money.

The last column is where the two TTLs change places. At 20 turns it is 1.91 minutes, so the 5-minute TTL almost never wins; at 5 turns it widens to 6.00. The shorter the session, the more the 5-minute TTL comes back.

Part two said sessions shorter than the crossover turn (2S / δ + 1, eight turns at these inputs) are finished by prefix caching alone. That figure is where token volume crosses over, not where price does. The table above replaces it with the price-based boundary — and a 5-turn session still favors rolling when the gap is under 11.1 minutes.

Put the two axes side by side. The 5-minute threshold moves 36.8 points with hit position and 14.0 points with turn count. Hit position is more than twice as large, and in an approval-gated agent it breaks structurally.

Axis 3 — tenant structure breaks sharing, not the threshold

The first two axes move where the threshold sits. Here, the assumption the threshold is computed under — that the prefix is shared across users — breaks instead.

Cache stampede. Right after a deploy, right after a prefix change, or right after traffic stopped and came back, there is no preface tape. With a hundred people arriving at once, each of them records the preface from the top. That is a cache stampede. The docs state that a cache entry only becomes available after the first response begins, and that if you need cache hits for parallel requests you should wait for the first response before sending the rest — so requests that depart together cannot read each other’s recordings. (prompt caching docs, retrieved 2026-08-23)

Concurrent requests	All write	One write + rest read	Multiplier
10	$0.63	$0.11	5.8x
100	$6.25	$0.56	11.2x
1,000	$62.50	$5.06	12.4x

Based on a 10,000-token preface, with each amount being the total across all the concurrent requests and output excluded. The mitigation is the docs’ own prescription: right after a deploy, send one request first to fill the cache and let the rest follow.

That changes what a failed deploy looks like, though. If the warming request fails or runs late, the deploy succeeds while the first wave of traffic pays 11.2x. Warming has to be a step in the deploy pipeline whose success you check.

Per-user profiles. With the profile in the system prompt, the preface tape splits per person — the same as writing someone’s name into the preface. The fix is to move the profile into the first user message. The cache prefix is assembled tools → system → messages, so once the profile is inside messages, everything ahead of it is shared again. At a 1,500-token profile, an 8,500-token shared prefix, and the first request of each of 100 users:

Placement	Cost	Saving vs. injecting into system
Profile injected into system	$6.25	—
Moved to the first user message, profile cached too	$1.41	4.4x
Moved to the first user message, profile not cached	$1.22	5.1x

Not putting a breakpoint on the profile is cheaper. The profile differs per user so it is never shared anyway, and caching it writes at 1.25x while not caching passes it at 1.0x.

When tools differ per tenant. Because tools sits at the very front of the prefix, a tool list that differs per tenant severs sharing across tenants. Inside a tenant the bytes are identical, so sharing still works — the scope narrows rather than vanishing. Building on the previous subsection’s move of the profile into the first user message, assume the 6,610 tokens of tool definitions differ per tenant while the remaining 1,890 (325 bash plus 1,565 system prompt — the premise table’s 3,065 minus the 1,500-token profile just relocated) is common. At one tenant the two columns are equal.

Tenants	Per-tenant tool sets as they are	Unified into a union
2	$1.27 (4.9x)	$1.59 (3.9x)
5	$1.42 (4.4x)	$2.70 (2.3x)
10	$1.66 (3.8x)	$4.54 (1.4x)
20	$2.15 (2.9x)	$8.23 (0.8x)

Leaving the per-tenant tool sets alone is cheaper on every row. Unify on the union and every tenant ships other tenants’ tool-definition tokens on every request, so at 20 tenants it reaches $8.23 — worse even than injecting the profile into the system prompt at $6.25.

So compare measured costs instead of converting

To measure a hit rate and compare it against a threshold, you have to find your turn count’s row and assume the hits are spread evenly through the session. The second assumption is wrong by as much as axis 1.

There is a route that needs no assumptions: build the session cost from your logs and compare it against what prefix-only caching would have cost. It takes five inputs.

1. Session ID — to group requests into sessions
2. Request timestamp — to get the gap from the previous request
3. Four usage fields — cache_read_input_tokens, cache_creation_input_tokens, input_tokens, output_tokens
4. S — the token count of your tools and system blocks. It is not recoverable from the usage fields, so count it directly
5. The TTL you are running

cost of the current setup, per session
  = cache_read_input_tokens      x $0.50
  + cache_creation_input_tokens  x that request's write rate
  + input_tokens                 x $5.00
  + output_tokens                x $25.00     (all per 1M)

the same session under prefix-only
  = NxS x $0.50 + history x $5.00 + output x $25.00
    (history = the sum of each request's input minus S)

If rolling is not on yet, reconstruct the left side from the logs. Take the gap from the previous request in the same session, call it a hit when the gap is under the TTL, and each request’s read and write tokens follow. That route uses measured gaps, so it needs no exponential assumption either.

Wherever the hits fell, whatever the gap distribution, whatever the turn count, this comparison holds. Run it on patterns with the hits scattered at random and it matches the model to the decimal.

Two organizations, run through the numbers

What separated the two examples is neither user count nor total traffic. It is the gap, the turn count, and whether the gap stays even through the session.

The three are not independent. A wider gap means fewer turns a person can fit into one session, fewer turns raises the threshold, and approval waits push the hits into the early turns. All three degrade together.

How to decide

0. First check that the preface tape is alive. The condition is a request rate, not a user count: above 0.2 per minute within the sharing scope (everything, or one tenant) it stays alive. Below that, even the breakpoint on the preface misses every turn, the cost becomes $4.30, and that is 6.2% more than the $4.05 of setting no breakpoint at all. Only then is the answer to turn caching off entirely. Splitting tenants across workspaces does not put you here as long as each tenant’s own request rate clears 0.2 per minute.

1. Compute the real per-session cost under both setups and compare them directly. The five inputs and two expressions above. Do not compare against a threshold.

2. Split cohorts by gap, because the answer splits by band. The figures below are the base case (20 turns) under the assumption that hits are spread evenly through the session. Prefix-only is $3.15 regardless of gap.

In-session average gap	5-min hits	1-hour hits	Rolling 5-min	Rolling 1-hour	Cheapest
2 min	91.8%	100.0%	$1.15	$1.13	rolling 1-hour
5 min	63.2%	100.0%	$2.00	$1.13	rolling 1-hour
20 min	22.1%	95.0%	$3.21	$1.37	rolling 1-hour
60 min	8.0%	63.2%	$3.63	$2.92	rolling 1-hour
90 min	5.4%	48.7%	$3.70	$3.63	prefix-only
180 min	2.7%	28.3%	$3.78	$4.62	prefix-only

The band where the 5-minute TTL wins is under 1.91 minutes at 20 turns. Unless it is an automated loop firing seconds apart, the 1-hour TTL is ahead.

What the split earns has to be on the table before anyone approves the work. Take a service that is half always-on (2 min) and half overnight (180 min), and price all three uniform choices against the split.

Choice	Always-on (2 min)	Overnight (180 min)	50:50 blend
Uniform, rolling 5-min	+2.5%	+20.1%	$2.47
Uniform, rolling 1-hour	optimal	+46.6%	$2.87
Uniform, prefix-only	+179.6%	optimal	$3.15
Split by cohort	optimal	optimal	$2.14

The minimum of the blend column is the split. The comparison is against the best uniform choice — rolling 5-min at $2.47 — not the worst, and the split comes in 13.4% under it. At $2,000 of implementation it pays back after 6,060 sessions. Below that many sessions a month, leave it on the best uniform setting and revisit later.

3. Send one pre-warming request right after each deploy. Across 100 users it turns $6.25 into $0.56. Check that the warming step succeeded.

4. Diagnose broken sharing from each session’s first request. Total write tokens grow linearly with user count whether sharing works or not, so the total alone makes it hard to tell which you are in. Look at cache_creation on the first request of each session: in a sharing service few such requests write the full S, while in a broken one nearly every session’s first request does. Watch that share.

5. Measure time to first token before turning rolling off on cost alone. With rolling off, every turn re-prefills the entire history — 57,000 tokens at turn 20, since the 10,000-token preface is read from cache and skips prefill. The price of keeping it on is (rolling − prefix-only) × monthly sessions, which for Example B is $294 a month. If the latency benefit beats that, keeping rolling on below the threshold is right — though that benefit accrues only on requests that hit, so it scales with the hit rate.

6. Put three values on a dashboard: the in-session gap distribution (not just its mean, but whether it skews early or late within a session), the turn count, and the rates. The first two move when the product does, and all three move the threshold. When any one crosses into the next row of a table above, re-run the direct comparison from step 1.

Part two said to use the 1-hour TTL in any band where human involvement can exceed five minutes. That recommendation compared the 5-minute and 1-hour TTLs on the premise that rolling is on, and never compared either against prefix-only. If the gap exceeds that row’s 1-hour tolerated gap, the premise fails and the comparison becomes rolling versus off. Part two also judged a 5-turn session not worth the engineering, since prefix-only alone already gives 37% savings and rolling only adds up to 52%; that judgment reverses too once the gap is in play — a 5-turn session favors rolling when the gap is under 11.1 minutes.

Every number here sits on the same example values as part two. Only the fact that the threshold is a function of the rates and the turn count alone holds regardless of those values; everything else is attached to a condition. Which is why this post’s conclusion is not a threshold but an instruction to stop using one. As long as a cache write costs more than list, caching that does not hit is a loss — and how big a loss is something you have to count.