Abstract
Indexer-based sparse attention reduces the cost of core attention by passing only a fixed, small number of important tokens to it. However, the indexer must still score the entire KV cache at every decoding step. This scoring overhead becomes a major bottleneck as the context length grows. We propose SPIN (Shadow Predictive Indexer) to reduce this indexer overhead. SPIN uses lightweight, history-based prediction to identify important KV blocks, avoiding the need to score the full KV cache at every decoding step. SPIN treats KV blocks and speculative decoding as first-class design and implementation considerations. Across extensive evaluations on long-context and agentic benchmarks, SPIN achieves 30-40% sparsity while preserving task quality. In end-to-end vLLM serving, SPIN improves output throughput by up to 14.9% and reduces median inter-token latency by up to 13.2%.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Fu, Y., Chang, J., Zhao, R., Long, B., Li, Y., Kamani, M., Jain, S., Raman, R., Safavi, T., Gupta, S., Fashi, P. A., Lee, M., Demouth, J., & Rouhani, B. D. (2026). SPIN: Shadow Predictive Indexer for Sparse Attention. https://omanscience.com/en/articles/spin-shadow-predictive-indexer-for-sparse-attention
MLA 9
Fu, Yao, et al. "SPIN: Shadow Predictive Indexer for Sparse Attention." https://omanscience.com/en/articles/spin-shadow-predictive-indexer-for-sparse-attention.
Chicago (author–date)
Fu, Yao, Jiahan Chang, Ritchie Zhao, Bryce Long, Yueying Li, Mahdi Kamani, Samkit Jain, Rahul Raman, Tara Safavi, Shreya Gupta, Parsa Ashrafi Fashi, Minseok Lee, Julien Demouth, and Bita Darvish Rouhani. 2026. "SPIN: Shadow Predictive Indexer for Sparse Attention." https://omanscience.com/en/articles/spin-shadow-predictive-indexer-for-sparse-attention.
Harvard
Fu, Y., Chang, J., Zhao, R., Long, B., Li, Y., Kamani, M., Jain, S., Raman, R., Safavi, T., Gupta, S., Fashi, P. A., Lee, M., Demouth, J. and Rouhani, B. D. (2026) 'SPIN: Shadow Predictive Indexer for Sparse Attention', Available at: https://omanscience.com/en/articles/spin-shadow-predictive-indexer-for-sparse-attention.
Vancouver
Fu Y, Chang J, Zhao R, Long B, Li Y, Kamani M, et al. SPIN: Shadow Predictive Indexer for Sparse Attention. https://omanscience.com/en/articles/spin-shadow-predictive-indexer-for-sparse-attention
IEEE
Y. Fu, J. Chang, R. Zhao, B. Long, Y. Li, M. Kamani, S. Jain, R. Raman, T. Safavi, S. Gupta, P. A. Fashi, M. Lee, J. Demouth, and B. D. Rouhani, "SPIN: Shadow Predictive Indexer for Sparse Attention," https://omanscience.com/en/articles/spin-shadow-predictive-indexer-for-sparse-attention.