الملخص
Long-context LLM serving is increasingly bottlenecked by decode, where large KV caches limit batch size and underutilize GPUs. Sparse KV cache offloading expands effective capacity by storing most historical KV blocks in CPU DRAM and recalling only selected blocks on demand. However, we find that existing offloading systems shift the bottleneck to CPU-GPU recall I/O: recall volume varies widely across layers, decode steps and requests, while headwise sparse selection fragments recalls into many small PCIe transfers. This paper presents PulseInfer, an I/O-centric sparse KV cache offloading system. PulseInfer hides variable recall latency with interruptible layer-wise scheduling, adapts offloading decisions with IO-Adaptive Offloading Admission, and coalesces fragmented transfers using SoloHead sparse selection and a gather-scatter I/O engine. Implemented on SGLang, PulseInfer improves decode throughput by up to 4.7x over SGLang and 2.6x over the best existing offloading baseline, while reducing TPOT by up to 76% and preserving near-lossless accuracy.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Zhang, Q., Zhou, K., Lu, K., Lu, H., Zhou, J., Su, Y., Bao, K., Wan, J., & Wu, F. (2026). PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding. https://omanscience.com/ar/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding
MLA 9
Zhang, Qiuyang, et al. "PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding." https://omanscience.com/ar/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.
شيكاغو (المؤلف–التاريخ)
Zhang, Qiuyang, Kai Zhou, Kai Lu, Haocheng Lu, Jian Zhou, Yuanpeng Su, Kun Bao, Jiguang Wan, and Fei Wu. 2026. "PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding." https://omanscience.com/ar/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.
هارفارد
Zhang, Q., Zhou, K., Lu, K., Lu, H., Zhou, J., Su, Y., Bao, K., Wan, J. and Wu, F. (2026) 'PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding', Available at: https://omanscience.com/ar/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.
فانكوفر
Zhang Q, Zhou K, Lu K, Lu H, Zhou J, Su Y, et al. PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding. https://omanscience.com/ar/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding
IEEE
Q. Zhang, K. Zhou, K. Lu, H. Lu, J. Zhou, Y. Su, K. Bao, J. Wan, and F. Wu, "PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding," https://omanscience.com/ar/articles/pulseinfer-i-o-centric-sparse-kv-cache-offloading-for-efficient-long-context-llm-decoding.