Preprint Open access
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each k …