Contents
Qwen3.8-27B (W4A16 AutoRound) on vLLM 0.27.1, one RTX 3090, 152k of context in a 22 GB slice, serving two agent workloads. One afternoon of measurements, one flag that mattered more than everything else combined, and two wrong theories worth naming.
The numbers first
| What we changed | Before | After | |
|---|---|---|---|
| Prefix caching off → on two requests, 21k shared prefix |
31.2 s | 4.1 s | 7.6x |
| Agent turn 1 → turn 2 same conversation, 60k → 62k tokens |
99.6 s | 6.6 s | 15x |
| Time to first token per prompt token 31 agent turns, aggregate |
3.286 ms | 1.624 ms | 2.0x |
| Same agent, alone vs sharing same client, ~90k prompt, four minutes apart |
6.6 s | 384.7 s | 58x worse |
| Prefill chunk 2048 → 8192 cost in KV cache |
3.6 GiB | 1.9 GiB | engine won't start |
| Context window, price of prefix caching | 196,608 | 155,648 | −41k tokens |
| Speculative draft depth 3 → 4 MTP, accepted tokens per draft |
1.96 | 2.22 | +13% decode |
What it does per second, once all of that is settled and nothing else is competing: ~9,500 tokens/s prefill and 25–28 tokens/s decode. Prefill is what you wait through before the first word appears; decode is the speed the answer arrives at after that. On an agent turn carrying tens of thousands of tokens of tool schemas, almost all of the waiting is the first one.
The rest of this is why.
One budget, spent in a fixed order
vLLM takes --gpu-memory-utilization, carves out the model weights, then activation scratch, then CUDA graphs, and whatever survives becomes the KV cache. The KV cache is last in the queue, and it is the thing every other setting quietly taxes.
Raising the prefill chunk looks like a scheduling change. It is not. --max-num-batched-tokens sizes the prefill batch, so a bigger value needs bigger activation scratch, and the KV pool absorbs the entire difference. At 8192, Qwen3.8's 196k window no longer fits:
ValueError: To serve at least one request with the model's max seq len (196608),
2.94 GiB KV cache is needed, which is larger than the available KV cache memory (1.92 GiB).
The cruelty is in the ordering. That check runs at the end of startup, after weight loading, after torch.compile, after graph capture — about four minutes in. Every wrong number costs a full boot before it tells you it was wrong.
There is a trap inside the trap. The flag appears in the process command line for the whole of that four minutes, so ps shows the new value on a process that is already doomed. An applied argument is not a started server. The check is the unit being active and Started server process in the log.
The 22 GB section
A 24 GB card in a desktop machine is not a 24 GB card. Wayland and the compositor hold a fluctuating 2–2.5 GiB, and vLLM refuses to start if its slice exceeds free VRAM at the instant it asks. The honest working figure is 22 GB, and the utilisation setting has to reflect it:
--gpu-memory-utilization 0.85 # not 0.93, which headless guides assume
The remaining headroom is not spare. The first request lazily allocates a sampler and speculative-decode workspace beyond the profiled budget — roughly 400 MiB the profiler never sees. At 0.88 that overflow took the engine down under live traffic. 0.85 leaves about 1 GiB of real slack, and that slack is what stops a desktop notification from killing your inference server.
What 22 GB buys for Qwen3.8-27B at 4-bit weights:
| Model weights | 13.97 GiB |
| Activation scratch, CUDA graphs, compile artifacts | ~2.5 GiB |
| KV cache (KVarN 4/2-bit dtype) | 2.5–3.6 GiB |
| Headroom you must not spend | ~1 GiB |
Every decision below is a fight over the third row.
Prefix caching is opt-in for hybrid models, and nobody tells you
This is the finding worth the whole exercise. Our agent turns were taking 137–154 seconds to first token on a 30k prompt, while the same assistant on llama.cpp answered in one second. The difference was not the engine's speed. llama.cpp was reusing the prompt prefix between turns and vLLM was not.
vLLM enables prefix caching by default — except for hybrid models, where it is held back while the feature matures:
default_prefix_caching = (
model_config.is_prefix_caching_supported and not model_config.is_hybrid
)
Qwen3.8 declares IsHybrid. So the default resolves to off, silently, and every turn of every conversation re-prefills its entire prompt. Nothing in the startup output says you are paying for this. You have to notice enable_prefix_caching=False in the config line, or that Prefix cache hit rate has sat at 0.0% since the day you installed it.
We spent an afternoon believing this was forced by the quantised KV cache. It is not — the engine runs kv_cache_dtype=kvarn_k4v2_g128 and enable_prefix_caching=True together without complaint. It was a default nobody had passed:
--enable-prefix-caching
Two caveats before you copy it. Qwen3.8 carries IsHybrid but not SupportsMambaPrefixCaching, so it lands in vLLM's align mode: reuse on block boundaries, not exact matches, and a share of every prompt is always recomputed. And reuse requires the prefix to match from the first token — a timestamp, a session id or a directory listing near the top of a system prompt breaks the cache there and everything after it re-prefills anyway.
What it costs
Prefix caching re-sizes the KV pages, and on a tight budget that is not free:
without: 3.54 GiB available · 2.94 GiB needed for 196k → starts
with: 2.51 GiB available · 3.02 GiB needed for 196k → refuses
About 1 GiB of KV, which at this page size is roughly 41k of context window. We took the trade — 196,608 down to 155,648 — because a window you rarely reach is worth less than a 15x cut on every turn you actually take. The engine prints its own ceiling if you want to find the edge:
Based on the available memory, the estimated maximum model length is 157696.
Concurrency is the enemy, not the goal
This is the part that inverted our assumptions. We had been sizing for four concurrent agents, reasoning that the engine serves several streams and independent work should overlap. The measurements say the opposite, and the cleanest pair is one client four minutes apart on the same prompt:
13:39:57 agent alone 6,623 ms 91,070 prompt tokens
13:44:30 agent sharing 384,705 ms 92,891 prompt tokens
Six and a half seconds against six and a half minutes, with one variable: whether anything else was asking at the same time. Not preemption either — that counter stayed at zero throughout.
Two causes compound. Prefill chunks interleave with the other stream's decode steps, so a prefill that runs at 9,500 tokens/s alone drops to a few hundred when sharing. And a 160k-token cache does not hold a 90k prompt and a 35k one alongside their own histories, so each arrival evicts the other's blocks and the reuse you just bought a context window for collapses to zero. Watched live, the split of prompt tokens by source reads 441 tokens/s recomputed and 0.0 from cache while both clients are in flight.
On a single card with a shared desktop, one agent at a time with a warm cache beats four agents sharing. The cache is a fixed resource and concurrency divides it.
Put the ceiling in front of the engine
A local engine does not refuse work it will serve badly. vLLM admits past its KV budget and then preempts, and a preempted sequence loses its blocks. So the ceiling has to sit in front of it: an admission gate that queues requests rather than letting them compete inside the scheduler.
max_concurrent: 4 # mirrors the engine's slot count
max_concurrent_tokens: 136000 # ~85% of "GPU KV cache size" from the startup log
queue_timeout_s: 300
A request count is almost meaningless on a long-context engine. Ours reports 160,779 KV tokens against a 155,648 window — 1.03 full-context requests — so a limit of eight streams is a ceiling it can never reach. The token budget is what actually protects the cache.
And if the box has one user, the right number is one. Several clients that do not know about each other — an editor, an assistant, a cron — are not a reason to serve them at once; on a single card they are a reason not to. Whoever queues still finishes sooner than both would by competing, and the gate is what turns that into policy rather than luck:
max_concurrent: 1
One accounting detail cost us twice the concurrency we should have had: charge the estimate for the completion a request will plausibly produce, not the one it declares. Our callers declared max_tokens: 65536 against a measured median of 769 tokens and a p99 of 8,948 — an overstatement of 20–60x. Sizing the context window still needs the declared figure; sizing occupancy does not, because the engine grows a sequence's blocks as it generates. Charging prompt + min(max_tokens, 8192) turned a lane that held one real agent turn into one that holds two.
The settings, together
--gpu-memory-utilization 0.85 # 22 GB, not 24: the desktop holds the rest
--max-num-batched-tokens 2048 # raising it is paid for out of the KV cache
--enable-prefix-caching # off by default on hybrid models
--max-num-seqs 8 # unreachable at this context depth; harmless
--kv-cache-dtype kvarn_k4v2_g128 # coexists with prefix caching
--block-size 128
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
max_model_len 155648 # what is left once caching takes its share
On the speculative line: at depth 3 the measured MTP acceptance ran 83% / 64% / 48% across draft positions, and a near coin-flip at the last position meant depth was the binding constraint rather than the model's agreement. Depth 4 accepts 81.6% / 61.5% / 45.4% / 33.4% — 2.22 accepted tokens per draft against 1.96, which is the +13% in the table. The engine warns that deeper drafts want a larger prefill chunk, and on this card that is the one thing the KV budget cannot afford.
What we would tell ourselves at the start
Read the startup log before changing anything. Available KV cache memory, GPU KV cache size and Maximum concurrency for N tokens per request are three lines that price every decision in this article, and they are printed on every boot.
Then price all the changes on paper before making any of them. They draw on the same pool and each wrong guess costs a four-minute boot to discover. We paid nine boots in one afternoon to learn three numbers that one calculation would have given us.
And measure time to first token at the engine, not at the proxy. A proxy's figure includes admission queueing and model loading; the engine's vllm:time_to_first_token_seconds is prefill alone. We spent an hour optimising a queue we had mistaken for compute, and another hour correcting the correction.
Jan Kozák and Claude