Abstract
Long-context LLM serving is increasingly bottlenecked by decode, where large KV caches limit batch size and underutilize GPUs. Sparse KV cache offloading expands effective capacity by storing most historical KV blocks in CPU DRAM and recalling only selected blocks on demand. However, we find that existing offloading systems shift the bottleneck to CPU-GPU recall I/O: recall volume varies widely across layers, decode steps and requests, while headwise sparse selection fragments recalls into many small PCIe transfers. This paper presents PulseInfer, an I/O-centric sparse KV cache offloading system. PulseInfer hides variable recall latency with interruptible layer-wise scheduling, adapts offloading decisions with IO-Adaptive Offloading Admission, and coalesces fragmented transfers using SoloHead sparse selection and a gather-scatter I/O engine. Implemented on SGLang, PulseInfer improves decode throughput by up to 4.7x over SGLang and 2.6x over the best existing offloading baseline, while reducing TPOT by up to 76% and preserving near-lossless accuracy.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhang, Q., Zhou, K., Lu, K., Lu, H., Zhou, J., Su, Y., Bao, K., Wan, J., & Wu, F. (2026). PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding. https://omanscience.com/en/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding
MLA 9
Zhang, Qiuyang, et al. "PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding." https://omanscience.com/en/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.
Chicago (author–date)
Zhang, Qiuyang, Kai Zhou, Kai Lu, Haocheng Lu, Jian Zhou, Yuanpeng Su, Kun Bao, Jiguang Wan, and Fei Wu. 2026. "PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding." https://omanscience.com/en/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.
Harvard
Zhang, Q., Zhou, K., Lu, K., Lu, H., Zhou, J., Su, Y., Bao, K., Wan, J. and Wu, F. (2026) 'PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding', Available at: https://omanscience.com/en/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.
Vancouver
Zhang Q, Zhou K, Lu K, Lu H, Zhou J, Su Y, et al. PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding. https://omanscience.com/en/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding
IEEE
Q. Zhang, K. Zhou, K. Lu, H. Lu, J. Zhou, Y. Su, K. Bao, J. Wan, and F. Wu, "PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding," https://omanscience.com/en/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.