A Safety Hook Truncated Every Code Generation

frequency_penalty=0.1 sounds harmless. On code it compounds until the model can't emit a newline, and it quits with finish_reason=stop so nothing looks wrong.

Summary

Our model started stopping mid-function on coding tasks. The cause was an anti-repetition measure we'd added to the gateway the day before: a frequency_penalty of 0.1 injected into every low-temperature request. Frequency penalty is count-based, and code repeats structural tokens hundreds of times, so the penalty compounds until the model literally cannot emit another newline or indent. It then stops — with finish_reason: stop, so nothing in the response indicates anything went wrong.

The report

"It stops mid-job when coding."

That's the whole bug report, and it's a good one — specific about the task type, specific about the failure shape.

The cause

We'd installed four anti-loop hooks in the gateway over the previous two days. Their job was to stop models getting stuck in repetition loops, which is a real problem worth solving.

One of them enforced a sampling floor on every non-exempt request with temperature ≤ 0.3 — which is all coding traffic, because coding agents ask for deterministic output. Part of that floor was frequency_penalty = 0.1.

Why that destroys code specifically:

Frequency penalty is count-based. Every time a token appears, its logit is penalized proportionally to how many times it has already appeared.

Prose has a naturally varied token distribution. Code does not. Code repeats structural tokens — newlines, indent runs, def, self, return, closing parens — hundreds of times in a single file, by design. Under a count-based penalty those tokens get progressively suppressed until the model can no longer emit them.

At which point it stops — not because it finished, but because continuing has become impossible.

And it reports finish_reason: stop. Not length. There is no truncation signal anywhere in the response. From the client's perspective the model simply decided a half-written function was complete.

That's what made this hard to see from the outside, and easy to see once measured.

The reproduction

Same prompt ("a 250+ line Python todo CLI"), temperature 1.0 explicit, reasoning off, max_tokens 8000, three runs each:

Output cut roughly in half, 3 out of 3.

The degenerate endings are worth describing, because they're diagnostic. The model stopped mid-docstring. It drifted into another language. One run ended with the model talking to itself: "try? No - just save and return".

Those aren't the signatures of a model that finished. They're the signatures of a model whose vocabulary is being closed off underneath it — a distribution getting squeezed until whatever's left is nonsense.

What wasn't the problem

Worth stating, because we suspected it first and it would have been the wrong fix.

The temperature floor itself — rewriting temp 0 to 1.0 with top_p 0.95 — is not the problem. Every clean run above was at temperature 1.0. The evidence for the floor's original purpose still stands.

It would have been easy to blame the temperature rewrite. It's the more visible, more objectionable part of the same hook, and it does cause a real problem elsewhere (part 1, where it contaminated our evals). But it wasn't causing this, and reverting it would have left the truncation in place while breaking the loop protection.

When one change bundles several modifications, isolate each one. We had two suspects in the same hook and only one was guilty.

A second bug found in the same code

While in there, we found the loop scrubber corrupting message history.

Its text-scrubbing function split on newline-following whitespace and rejoined with " ".join. That strips indentation and newlines from any assistant message it processes — so replayed code in conversation history came back flattened into a single line.

A model reading its own flattened code as context is going to produce worse code. That's a quiet, compounding quality problem with no error message at all.

The fixes

• Removed the frequency penalty injection. The floor is now temperature and top_p only.
• Replaced it with an evidence-gated presence_penalty of 0.2. Presence penalty is binary — a token is penalized once for appearing at all, not repeatedly by count — so it doesn't compound on structural repetition. And gating it on actual evidence of a loop means normal generation never sees it.
• Rewrote the scrubber byte-preserving and code-fence-aware.

Post-fix reproduction at temperature 0: 2/2 clean, 3871 and 3531 tokens, proper if __name__ == "__main__" endings, and the gateway log confirms no penalty was applied.

The general shape

This is a mitigation causing a worse problem than the one it prevents, and there's a recognizable pattern to it.

A global sampling modification cannot be correct for all content types. The parameter that suppresses prose repetition suppresses code structure, because in one case repetition is a defect and in the other it's the syntax. Any penalty applied uniformly across a mixed workload will be wrong for part of it.

Prefer evidence-gated mitigations to always-on ones. "Apply a penalty when we detect a loop" is strictly better than "apply a penalty always, in case there's a loop." The detection cost is trivial next to a permanent quality tax on every request.

Know the difference between count-based and presence-based penalties. They sound interchangeable and behave completely differently on structured output. Frequency penalty compounds; presence penalty doesn't.

Watch for mitigations that mask their own damage. finish_reason: stop on a truncated generation is the worst property of this bug. If the hook had produced length, or logged when it applied, we'd have found it the same day it shipped.

Key takeaways

1. frequency_penalty is count-based and compounds on code, where structural tokens repeat hundreds of times by design. Use presence_penalty if you need something.
2. A penalty-truncated generation reports finish_reason: stop. There is no truncation signal. Check output length against a control, not the finish reason.
3. Isolate each modification in a bundled change. We had two suspects in one hook; only one was guilty, and reverting both would have been wrong.
4. Degenerate endings are diagnostic. Mid-docstring stops, language drift, and self-talk mean a distribution being squeezed, not a model finishing.
5. Gate mitigations on evidence. Always-on protection against an occasional failure is a permanent tax on every request.
6. Log when a hook modifies a request. Ours didn't, which is why a one-day-old change took a reproduction campaign to find.