Qwen3.8-27B, 180k context on one RTX 3090: two 90k conversations with warm caches, and a 2.4x faster kernel that lost

Written by Claude with Jan Kozák, from measurements on Jan's workstation. "We" means the two of us: Jan asked the questions and made the calls, Claude ran the benchmarks and wrote it up.

If you serve a hybrid long-context model such as Qwen3.8-27B from vLLM on one 24 GB card, and more than one agent wants it, this is for you. You get the measured answer to "can two long conversations share the card without destroying each other's prefix cache", the admission setting that gives you two conversations and the full window from one loaded model, and a benchmark trap that made a 2.4x faster kernel the wrong choice.

What changed since the last post

The last post ended on a rule: on a single card, one agent at a time beats two sharing it. That rule was measured on vLLM 0.27.1 with a 160,779-token KV pool. The same model on the current stack holds 210,828, and the rule no longer holds.

Qwen3.8-27B W4A16, one RTX 3090	Now	Last post
vLLM stack KVarN 4/2-bit KV, MTP drafts	0.30.0	0.27.1
KV pool "GPU KV cache size" at startup	~210k 203,372–210,828	160,779
Context window, one caller	180,224	155,648
Second agent arrives at ~90k prompt tokens served from cache	92–95% warm turns 5–21 s	0% 6.6 s → 384.7 s
Concurrency setting	2 while they fit ~190k	one request

The rest of this is how we measured it, what to set, and why the faster engine we tried on the same night is not the one we run.

Two 90k conversations, both caches warm

The test: two conversations start at the same moment against the same engine, three turns each. Every turn adds about 2,300 tokens of new input and a 300-token answer, roughly how an agent conversation grows. The engine runs with --enable-prompt-tokens-details, so every response reports how many prompt tokens came from the cache. No other traffic was on the card.

Size per conversation	Turns 2–3 cached share	Turns 2–3 TTFT first / second	Turn 1 (cold) TTFT first / second	Decode per stream
~60k + ~62k	89–93%	4.5–5.6 s / 14.7–16.6 s	40 s / 87 s	42–50 tok/s
~80k + ~72k	93–94%	6.0–6.2 s / 15.5–18.4 s	58 s / 117 s	32–48 tok/s
~91k + ~90k	93–94%	6.1–6.3 s / 21 s	74 s / 154 s	28–41 tok/s

At 181k tokens resident in a 211k pool both conversations keep their prefix, turn after turn. The second conversation's time to first token is always the larger one because its prompt is read after the first one's, which is also why its cold turn takes twice as long: prompt reading is serialised, only decoding overlaps. Each stream decodes slower than it would alone (37–56 tok/s at those depths), and that is the price. What you buy is that the second agent starts in seconds instead of waiting minutes behind the first.

We then ran the same test through the proxy that fronts the engine, with two client identities: two ~80k conversations, admitted together (two in flight, none queued), 92–95% cached on turns 2 and 3, warm time to first token 5–18 s.

Why the old measurement said the opposite

Nothing about the model changed. The pool did. The old measurement put a ~90k prompt beside a ~35k one in a 160k cache: 125k of live prompt, plus both conversations' growth, plus the recurrent state a hybrid model keeps per sequence, in a pool that could not hold all of it. Every arrival evicted the other's blocks, and the cache served nothing. Between the two posts the stack moved to vLLM 0.30 with a newer KVarN port, and the KV memory was pinned to a fixed 3.4 GiB instead of a share of VRAM. Together they took the pool from 160,779 to about 210,000 tokens; we did not isolate which change bought how much. Two 90k conversations now fit with room to grow.

So the rule from last time was right about its pool and wrong as a rule. The question to ask your engine is not "how many requests" but "how many tokens of conversation fit at once". Its startup log answers it: GPU KV cache size: N tokens. Two conversations fit when their summed context stays a few thousand tokens under N.

One window for one caller, two for two

Jan's condition for this was that a single long conversation must not lose the 180k window: 90k each is fine for two agents, and not enough for one big one. Two model entries with different launch flags would mean a vLLM restart, minutes, every time traffic switched between them. A token-gated admission lane in front of the engine gives both from the one loaded model:

max_concurrent: 2             # two requests in flight at most
max_concurrent_tokens: 190000 # ...while their summed estimate fits
                              # a request larger than this is admitted alone

Two conversations run side by side while their estimated context adds up to 190k or less. A single 170k request arrives, finds the budget too small for it plus anything else, and is admitted alone with the full window. The budget is set under the smallest pool this card has ever produced at startup (193,732 tokens), because the pool moves with whatever the desktop holds in VRAM at load time. Our proxy estimates tokens as characters divided by four, which under-counts code, so the budget can occasionally admit a pair that does not quite fit. When that happens vLLM preempts one and recomputes it later. That is slow, and nothing crashes.

A 2.4x faster kernel that lost

The same evening we tried the current HyperQwen release: vLLM 0.32.0 with 54 patches, built beside production on the same card. Its plain upgrade, with our flags, measured the same as what we run. Its new long-context arm, int8 KV read by a custom attention kernel (PTH_FA=1), looked like the obvious win:

KVarN keeps a big window small in memory and pays for it on every token it writes, so decoding slows as the prompt grows: 94 to 37 tok/s. The int8 arm barely slows: 125 to 89. Quality held (GSM8K 0.965 against 0.960) and tool calls worked. It costs window, 143,520 tokens of pool against about 210,000, and we were ready to take that trade for a 2.4x faster answer at 106k.

Then we looked at whole agent turns instead of decode rates. Both engines read a prompt at the same speed, about 1,100–1,700 tokens per second, so on a turn that starts from scratch the int8 arm wins only the writing:

One agent turn at ~106k, 800-token answer	Current stack	int8 kernel
Cold: whole prompt read	89 s + 22 s = 111 s	98 s + 9 s = 107 s
Warm: prefix from cache, ~2k new tokens ~6 s to first token, as measured warm at ~90k	~6 s + 22 s = ~28 s	~6 s + 9 s = ~15 s, if it cached

The int8 arm's whole gain lives in warm turns, and on this card it never had one. With prefix caching enabled it reported Prefix cache hit rate: 0.0% for an entire boot, and every follow-up turn re-read its whole prompt. While one request was reading its prompt, the other's decode also dropped to about 7 tok/s, against 48 on the current stack. We did not find the cause: nothing in the log warns, and the project's own documentation lists prefix caching on for this arm. Until it caches, a benchmark that reports decode speed is measuring the half of the turn that matters least.

Reading the cache numbers right

Two things about measuring this nearly misled us.

vLLM does not report per-request cache hits unless you ask. Without --enable-prompt-tokens-details the usage block of every response leaves out prompt_tokens_details.cached_tokens, and nothing warns. Our proxy had a column for it that was empty on every request. The engine-wide ratio was there all along in Prometheus (vllm:prefix_cache_hits_total over vllm:prefix_cache_queries_total, 90.6% for the past week), but it cannot tell you which caller benefits, or what a new sub-agent's first turn reuses. The flag costs no memory.

On a hybrid model, about 94% cached is a full hit. Qwen3.8 mixes linear-attention layers with attention, so vLLM can only resume a cached prefix where it saved the recurrent state, at fixed block boundaries. Every cached count we saw was an exact multiple of 2,048, and each warm turn re-reads its new tokens plus the tail chunk. Our first pass bar was "at least 90% cached", and it called healthy turns at 89.1% failures. Judge it by warm time to first token against cold instead: seconds means the cache held, a minute or more means it was evicted.

What to do on your card

1. Read your pool. Find GPU KV cache size: N tokens in vLLM's startup log, over several boots, and take the smallest.
2. Gate by tokens, not requests. Put an admission budget in front of the engine a few thousand tokens under that minimum, allow two requests, and let anything larger than the budget run alone. If N is under about twice your typical conversation, keep it at one: that is where the old rule still applies.
3. Turn on --enable-prompt-tokens-details and watch warm turns' time to first token. Seconds means it works.
4. Before switching engines for decode speed, measure a warm turn. A faster kernel that does not hit the prefix cache loses to a slower one that does.

Sources

• Qwen3.8-27B with 152k context on 22 GB: one flag, 15x faster agent turns, the previous measurements on this card
• syv-ai/HyperQwen, the patched vLLM stack, and its vLLM 0.32.0 port notes
• vLLM releases
• Qwen/Qwen3.8-27B model card

Every figure above comes from one sitting on one RTX 3090 at its 300 W preset, with the desktop holding about 2.3 GB of VRAM, one boot per configuration. We did not test other cards.

Jan Kozák and Claude