Every Failure Silently Scored 0.5

Running a third-party eval framework against our own models. Four traps, one of which produces a complete, plausible results table built on zero model calls.

Summary

We wired an open-source LLM-as-a-verifier framework to our self-hosted models to reproduce its published best-of-N results. It worked, eventually. Getting there meant finding four traps — the worst of which is that every failure path in the codebase degrades to a 0.5/0.5 tie instead of raising, so an authentication failure produces a complete, plausible, entirely fabricated results table.

Trap 1: the silent 0.5

Every failure path in the framework degrades to [0.5, 0.5, 0.5] instead of raising. The scoring function swallows exceptions; model resolution can throw a 401 inside it.

What you get is a results table with numbers in every cell, appropriate-looking magnitudes, and no indication that zero model calls were made.

This is the most dangerous bug shape in evaluation software. A crash costs you an hour. A plausible fabricated result costs you a decision.

The check, and it takes ten seconds:

assert token_usage()['calls'] > 0

Run the framework's own quickstart first and confirm the call counter moved. Do that before any real run — and again every time you change credentials, endpoints, or environment.

Generalizing: any eval framework with a neutral default score needs a "did it actually run" assertion. 0.5 for a tie, 0 for a failure, empty string for no answer — all of these look like data.

Trap 2: .env can't override your shell

The framework's dotenv loading uses os.environ.setdefault. Our shell profile exports a real API key for a different service.

So setdefault finds a key already present, does nothing, and the framework authenticates against our endpoint with the wrong credentials. 401. Which lands in trap 1 and produces a beautiful table of 0.5s.

setdefault semantics mean the shell always wins. If you've ever wondered why your .env isn't taking effect, check whether the loader uses setdefault or assignment — they're one word apart and opposite in behavior.

We fixed it with a wrapper script that exports the right variables into the command environment explicitly, rather than trusting file-based config to win.

Trap 3: a pinned model that wasn't pinned

The first real run came in at 234 seconds per call — a 27-hour ETA.

The framework was pointed at a load-balanced alias, and every call was landing on one cluster that happened to be in scheduling collapse (2 running, 2 waiting, 0.4–0.9 tok/s) while the other sat completely idle.

An idle engine on the machine you're logged into is not evidence the run is stuck locally. Check every backend.

We pinned to a specific cluster and got 11.0 seconds per call — a 21× difference that had nothing to do with the model or the framework.

Then a subtler version of the same problem: our gateway has a shape router that classifies any request above a size threshold as "heavy" and pins it to a dedicated two-slot lane, overriding the requested model alias.

Every prompt in this benchmark is 500,000–720,000 characters. So 100% of calls landed in that shared lane regardless of what we pinned. The pinning was a no-op and the logs said so — the request logged as HEAVY lane=<other cluster> while the alias said something else entirely.

That obfuscation is working as designed; it keeps enormous requests off the cluster serving interactive traffic. But it makes this particular benchmark untestable through the gateway, and the conclusion is that the workload doesn't belong there. Either bypass the gateway with a direct tunnel, or run the verifier against a hosted model.

When a proxy can override your routing, your routing config is a request, not an instruction. Read the proxy's decision log to find out what actually happened.

Trap 4: inherited concurrency defaults

The benchmark ships with max_workers = 2000.

At ~80,000-token prompts, that is precisely the shape that wedges an engine. We know, because we've done it.

Never inherit a concurrency default from a benchmark repository. It was chosen for someone else's hosted API with someone else's rate limits, and it's the single easiest way to take down your own infrastructure while trying to measure it.

What we confirmed empirically

Since the whole point was checking whether our stack could run this class of workload, the compatibility findings are the actual result:

• logprobs with top_logprobs: 20 survive both the gateway and speculative decoding. All 20 alternatives come back. This was the main open question — speculative decoding is exactly the sort of feature that could quietly break logprob reporting.
• The prefill path passes through the gateway intact — continuing a final message, suppressing the generation prompt, and constrained choice outputs all work end to end.
• The model puts its probability mass on the score letter with a leading space (' T' rather than 'T'). The framework already allows both spellings, but any fork that hardcodes the letter list needs to keep that.
• Prefix caching measured 76.7% on a real trajectory pair, closely matching the paper's reported 78.4%.
• A real swing-task pair scored correctly — passing trajectory 0.992, failing trajectory 0.779.

One honest gap: our separation is weaker than the paper's hosted model (0.591/0.457/0.452 against their 0.731/0.384/0.384). The ordering is right, the confidence is lower. For a best-of-N verifier, ordering is what matters — but narrower margins mean more ties, and that's a real difference worth reporting rather than rounding away.

Read what the benchmark actually runs

Worth checking before you budget time for one of these.

The "self-verification" section of this benchmark runs no agent at all. It ships 445 pre-recorded trajectories, and the scripts only call a model to score them. Only the swing tasks — where the outcome could go either way — are actually verified: 24 of 89 for best-of-3, 36 of 89 for best-of-5.

That's a completely legitimate experimental design, and it's much cheaper than it sounds if you were budgeting for agent runs. But "we reproduced the benchmark" means something quite specific here, and it's worth knowing which thing you reproduced.

We also found a documentation bug: one script's docstring claims two pivots and the code sets one. Use the general entry point with explicit flags rather than the convenience script.

A harness that silently scores every failure as 0.5 will make any model look fine — worth catching before you trust a benchmark, which is the point of the setup audit.

Key takeaways

1. Assert that model calls happened. A framework that degrades failures to a neutral score will hand you a complete fabricated results table.
2. load_dotenv with setdefault means your shell wins. Export credentials in the command environment rather than trusting a file.
3. Check every backend before diagnosing a slow run. An idle engine on your own machine says nothing about where the traffic went.
4. A proxy that can override routing makes your model pin a suggestion. Read its decision log.
5. Never inherit a benchmark's concurrency default. Ours shipped 2000 workers.
6. Read what the benchmark actually executes. Ours scores pre-recorded trajectories and verifies 24 of 89 tasks — cheaper than expected, and a narrower claim than "we ran the benchmark."
7. Report weaker-than-published separation rather than rounding it away. Right ordering with lower confidence is a real, reportable difference.