Notes from the machine room.
What I learn building production systems and the machinery around language models. Written when there is something worth writing.
What I learn building production systems and the machinery around language models. Written when there is something worth writing.
Last time we wrote that on one RTX 3090 a second agent wipes the first one's prefix cache (6.6 s turned into 384.7 s). On the current vLLM stack the KV pool holds 210k tokens, and two ~90k Qwen3.8-27B conversations share the card with both caches warm: 92–95% cached, follow-up turns in 5–21 s. One caller still gets the full 180k window. A new int8 kernel decoded 2.4x faster at 106k but never hit the prefix cache, so it lost. Read GPU KV cache size from your startup log and gate concurrency by tokens, not by request count.
A remote TCP bug scored 9.8 went public on 15 October with its fix. Ubuntu signed fixed kernels for 26.04, 24.04 and 22.04 on 23 October, but they sit in -proposed: every released LTS kernel is still vulnerable, and apt upgrade will not fix it. To fix it now, install the four kernel packages for your release from -proposed and reboot. Until you can reboot, drop incoming TCP that carries IPv4 options with one nft rule. Both are below, ready to copy.
Prefix caching is off by default on hybrid models and nothing tells you. What that costs, what turning it on costs back, and why running four agents at once is slower than running one.
We use Google Analytics to count visits. Nothing else is stored.