The Benchmark Said 18%. Production Said 0.07%.

A benchmark found a serious failure rate in our serving stack. We nearly built a gateway routing layer to fix it. Then we measured real traffic.

Summary

A benchmark told us that 18.3% of requests to our LLM stack returned an empty answer. That's a severe number, and it triggered a design for per-request budget routing at the gateway. Before building it, we measured the same condition on real traffic: 0.07%, one request in 1,520 over thirty days. The 18.3% was an artifact of how the benchmark set token budgets. What was broken was our ability to see the difference — and that turned out to be a one-line health check default.

Part 4 of a series on serving a hybrid-reasoning model in production.

The alarming number

Context: we'd just enabled reasoning mode as a server default to fix an accuracy regression in negative judgments (part 1). A benchmark run immediately exposed the other edge of that decision.

Twenty scenarios, three repeats, on one cluster: 15 of 60 items hit finish_reason=length, and 11 of 60 (18.3%) returned empty text with no tool call. Mean score on truncated items was 0.257 against 0.944 on the rest.

It wasn't stochastic. Five scenarios failed on all three repeats — the model spending its entire budget in the reasoning trace and never emitting an answer. Two scenarios burned all 12,288 tokens without producing a single character of output.

This is the failure mode from part 3: reasoning-on with a budget below roughly 3,200 tokens returns content: None, no error raised. The caller gets an empty string.

For scale: it's about 20× larger than a parser bug we'd spent real effort driving from 5–8% down to 0.8%. And it hands the caller the identical symptom.

The design that didn't get built

The conclusion looked forced. There is no globally correct setting:

• Reasoning off → yes/no accuracy collapses (71.7% → 61.0%, the no bucket 75% → 39%)
• Reasoning on → empty answers whenever a caller's budget is tight

So resolution has to be per-request. A gateway policy keyed on requested max_tokens: if the caller asks for fewer than ~3,200 tokens, force reasoning off; otherwise leave it on. Maybe 150 lines in a gateway hook, plus a config surface, plus tests, plus a new thing that can break at 3am.

We wrote the design. We didn't build it. First we went to look at what production actually does.

What production actually does

The test is simple enough to describe in a sentence: does completion_tokens pile up against a round max_tokens ceiling?

If callers are getting truncated, you'll see a spike of completions sitting at exactly 1024, or 2048, or whatever the client library defaults to. Truncation leaves a fingerprint.

Over thirty days and 1,520 requests: one. 0.07%.

No pile-up at any round number. The maximum completion observed was 17,542 tokens — a client that let the model run as long as it needed.

The reason is that our real callers aren't benchmark harnesses. They're agent SDKs, and agent SDKs don't set tight token budgets, because their whole job is to let a model work through a multi-step task. The benchmark set conservative per-scenario budgets because that's what a benchmark does — it's controlling a variable. That control was the artifact.

Decision: keep reasoning on, build no budget routing. Writing a per-request gateway policy to address a 0.07% condition is optimizing against the instrument.

The thing that was actually broken

Here's the part worth keeping.

The reason we couldn't immediately answer "how often does this happen in production?" was that our monitoring had been reporting finish_reason=length on 98.1% of requests. A number that high carries no signal — it can't distinguish a healthy stack from a broken one.

The cause: the gateway's background health check defaulted to max_tokens=16. With reasoning on, the model spends all sixteen tokens thinking. Every health check truncated, by construction, forever.

Our monitoring was measuring itself.

Raising it to BACKGROUND_HEALTH_CHECK_MAX_TOKENS=1024 dropped the length share from 98.1% to 24.6%, and the remainder is a separate watchdog probing at max_tokens=8.

We tried 256 first. Not enough — health-check reasoning on the prompt "What's 1 + 1?" spans 48 to 440 tokens. Which is its own small finding: the model sometimes reasons for hundreds of tokens on a trivial liveness prompt. That's a sighting of a known runaway-reasoning issue, caught incidentally by a config fix.

Two lessons stacked on top of each other. A health check that probes with an unrealistic parameter doesn't just fail to detect problems — it actively poisons the aggregate metrics that would have shown you the problem. And the fix wasn't code, it was a default nobody had looked at since the stack was reasoning-free.

What we built instead

Not a router. An alarm.

A daily job at 03:50 that measures real-traffic truncation and pages if it exceeds 3%, against a measured baseline of 0.07%.

That alarm — not a benchmark number — is the trigger to revisit the decision. If our traffic mix changes and clients start setting tight budgets, we'll find out from production, and then the routing layer becomes worth its complexity.

This is the trade we'd make again: a threshold alert on the real signal costs an afternoon and can't break the request path. The routing layer would have been permanent complexity purchased against a number that turned out to be measurement noise.

Key takeaways

1. Benchmarks control variables; production doesn't. A benchmark's token budget is a deliberate control, and it can manufacture exactly the failure mode you're testing for. Check whether the artifact-producing parameter matches reality before you act on the result.
2. Truncation leaves a fingerprint. Completions piling up on a round number is a one-query check for "are my callers getting cut off." It converted a scary benchmark result into a decision in about an hour.
3. A metric at 98% carries no information. If a monitoring number is pinned near a limit, suspect the instrument before you interpret the value.
4. Health checks should probe with realistic parameters. Ours used max_tokens=16 because that was sensible before the stack had reasoning mode. It silently invalidated an entire metric afterward.
5. Prefer an alarm on the real signal to a system that prevents a hypothetical. The routing layer was permanent complexity. The threshold alert took an afternoon and will tell us if the situation ever changes.
6. Write the design, then go measure. We had the routing design fully specified. Thirty minutes of looking at production data retired it. That order — design, then verify the premise — is cheap and catches this class of mistake.