Our GPU Cluster Is Idle 97.4% of the Time

Thirty days of gateway logs, one SQL sweep, and the number that retired a 29% throughput upgrade before we built it.

Summary

We measured the concurrency distribution of our self-hosted LLM cluster across thirty days: 74% of busy time runs at three concurrent requests or fewer, and total busy time is nineteen hours out of a possible thirty days. A 2.6% duty cycle. That single number retired a throughput upgrade that had measured 29% better in someone else's benchmark, and it's now the first thing we check before starting any performance project.

The measurement

Thirty days of gateway spend logs, swept into a running concurrency sum. As a share of busy time — idle excluded, so this describes what happens when the cluster is actually doing something:

Three or fewer: 74.42%. Four or more: 25.58%.

And the headline: total busy time was 68,426 seconds — 19.0 hours across 30 days. A 2.6% duty cycle.

Request volume is healthy, 3,000 to 10,700 per day. The requests are just short and they rarely overlap.

What this kills

Any optimization that only pays off at high aggregate concurrency is close to worthless on a cluster like this. That's most of the interesting ones.

It retired a real candidate. An upstream project measured a different vLLM version beating ours by roughly 29% aggregate throughput, which read like the single biggest lever available to us. Upgrading was queued as the next major piece of work.

Then we looked at where that 29% comes from. It's band-specific:

• At concurrency 1: a tie.
• At concurrency 2: our runtime actually wins by 10.8%.
• At concurrency 4+: their build leads, and that's where the aggregate number comes from.

Weighted by our actual distribution — 74% of busy time at c≤3 — the upgrade is worth about +5%. Against that: it would have cost us a patch that took JSON-mode validity from 8/24 to 24/24, on a cluster that is idle 97.4% of the time.

Closed as not worth doing. That decision took an afternoon of SQL and saved a multi-week migration whose benefit was mostly imaginary at our traffic shape.

The trigger to revisit is sustained busy time at c4+, not request count. Request count going up doesn't change the answer. Requests overlapping does.

Health checks outnumber real traffic 60 to 1

Before you can measure anything, you have to get the pollers out.

98,639 logged requests over thirty days. Roughly 1,520 of them were real.

Everything else was health checks, watchdog probes, and local benchmarks. Any query against the logs that doesn't exclude them is measuring the monitoring system:

AND request_tags::text NOT LIKE '%health-check%'
AND request_tags::text NOT LIKE '%curl%'      -- watchdog, max_tokens=8
AND request_tags::text NOT LIKE '%urllib%'    -- local probes / benchmarks

An honest caveat on our own table above: the concurrency distribution is health-check shaped for the same reason. The "not throughput-bound" conclusion holds a fortiori — real traffic is smaller and sparser still — but the per-level percentages aren't a clean read of user traffic. We're publishing the number we'd act on, with the qualification attached, rather than a cleaner number we didn't measure.

Real clients turn out to be agent SDKs, and they behave very differently from benchmarks: one at 552 requests averaging 2,023 output tokens with a maximum of 17,542; another at 629 requests averaging 337; a coding agent at 144 requests averaging a 42,808-token prompt.

Nobody sets tight token budgets. They let the model run. That fact retired a second project — see part 4, where a benchmark's 18.3% empty-answer rate turned out to be 0.07% in production for exactly this reason.

The fake 100-second latency tail

While measuring time-to-first-token from the same logs, we produced a scary number: a p50 TTFT of 102 seconds in the 16–32K prompt band, and bad p95s everywhere.

Both artifacts.

completionStartTime is only a real first-token time for streaming requests. For non-streaming ones the gateway sets it to approximately endTime, so completionStartTime - startTime is the total generation time wearing a TTFT label.

The obvious filter doesn't work:

-- WRONG: lets non-streaming rows through
AND "completionStartTime" <> "endTime"

-- RIGHT:
AND EXTRACT(EPOCH FROM ("endTime" - "completionStartTime")) > 1.0

The gap on a non-streaming row can be a millisecond rather than exactly zero. Inequality passes it.

Filter with a tolerance and the picture inverts completely: real streaming TTFT is 2.3–2.5 seconds and the cluster has no first-token latency problem at all.

Two near-misses in one afternoon — a fake latency crisis and a real upgrade decision — both turning on how a timestamp column is defined.

The SQL, and the two gotchas

The sweep is a standard start/end event fold:

WITH ev AS (
  SELECT "startTime" t,  1 d FROM "LiteLLM_SpendLogs" WHERE ...
  UNION ALL
  SELECT "endTime"   t, -1 d FROM "LiteLLM_SpendLogs" WHERE ...
),
run AS (
  SELECT t, sum(d) OVER (ORDER BY t, d DESC ROWS UNBOUNDED PRECEDING) c
  FROM ev
),
seg AS (
  SELECT c, EXTRACT(EPOCH FROM (lead(t) OVER (ORDER BY t) - t)) secs
  FROM run
)
SELECT c, sum(secs) FROM seg WHERE c > 0 GROUP BY c ORDER BY c;

Two operational notes that cost us time:

The gateway's Postgres role is not what you'd guess from the service name. Use the container's own environment rather than hardcoding a username: docker exec <db-container> sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB ...'.

Write the SQL to a file and copy it in. Quoting a query through nested SSH mangles it in ways that produce syntax errors far from the actual problem.

Spend-log writes lag minutes behind. We once concluded a fresh test request had routed to the wrong backend, because the newest row in the table was actually an older request from somewhere else. Never attribute a just-sent request from spend logs alone — use a router decision log, or watch the engines' live metrics counters.

Why this is now step zero

We keep this measurement as a standing reference and check it before starting any performance work. Not because 2.6% is interesting on its own, but because it converts "would this optimization help?" from an argument into a lookup.

The useful question about a benchmark result isn't whether it's true. It's whether the regime it was measured in is the regime you run in. A 29% win at concurrency 8 is a real number that describes a cluster we do not operate.

Key takeaways

1. Measure your duty cycle before optimizing throughput. Ours is 2.6%. Most throughput work is worth approximately nothing at that shape, and one query tells you.
2. Weight external benchmark results by your own concurrency distribution. A 29% aggregate win became +5% for us, and turned negative once we counted what adopting it would cost.
3. Health checks can outnumber real traffic 60:1. Filter them explicitly, or every conclusion you draw describes your monitoring.
4. Know which of your timestamp columns mean what for streaming vs non-streaming. One misread column produced a 100-second latency crisis that didn't exist.
5. Filter float-ish equality with a tolerance. a <> b let millisecond-gap rows through and poisoned the whole distribution.
6. Set a revisit trigger in the right units. Ours is sustained busy time at c4+, not requests per day. Volume can double without changing the answer.