الملخص
Long-context inference has become a defining capability of large language models, but exact dense attention remains costly due to its quadratic scaling with sequence length. Block-sparse attention offers a hardware-friendly alternative by routing each query block to a small set of relevant key blocks, yet accurate training-free block routing remains difficult. Existing routers often pool post-RoPE token representations, which entangles semantic aggregation with RoPE-induced geometry and attenuates local positional cues through high-frequency phase cancellation. To resolve this mismatch, we propose \textbf{Semantic-Geometric Decoupled Routing}, a training-free block routing framework that shifts semantic aggregation to the pre-RoPE space and reconstructs geometric bias with an offline structural prior and relative block distances. This decomposition yields an explicit closed-form block routing score without token-level search or post-hoc calibration. Experiments on long-context text and video tasks show that our method approaches full-attention accuracy across 4K--128K contexts, keeps routing overhead below 3.4 ms, and achieves a 5.03$\times$ speedup over FlashAttn at a 128K context length.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Long, X., Sun, W., Gao, W., Jiao, P., Qi, B., Zhu, F., Zhong, Y., Hoi, S., & Zhou, B. (2026). Block-Sparse Attention with Semantic-Geometric Decoupled Routing. https://omanscience.com/ar/articles/block-sparse-attention-with-semantic-geometric-decoupled-routing
MLA 9
Long, Xinwei, et al. "Block-Sparse Attention with Semantic-Geometric Decoupled Routing." https://omanscience.com/ar/articles/block-sparse-attention-with-semantic-geometric-decoupled-routing.
شيكاغو (المؤلف–التاريخ)
Long, Xinwei, Weigao Sun, Weibo Gao, Pengkun Jiao, Biqing Qi, Feida Zhu, Yiran Zhong, Steven Hoi, and Bowen Zhou. 2026. "Block-Sparse Attention with Semantic-Geometric Decoupled Routing." https://omanscience.com/ar/articles/block-sparse-attention-with-semantic-geometric-decoupled-routing.
هارفارد
Long, X., Sun, W., Gao, W., Jiao, P., Qi, B., Zhu, F., Zhong, Y., Hoi, S. and Zhou, B. (2026) 'Block-Sparse Attention with Semantic-Geometric Decoupled Routing', Available at: https://omanscience.com/ar/articles/block-sparse-attention-with-semantic-geometric-decoupled-routing.
فانكوفر
Long X, Sun W, Gao W, Jiao P, Qi B, Zhu F, et al. Block-Sparse Attention with Semantic-Geometric Decoupled Routing. https://omanscience.com/ar/articles/block-sparse-attention-with-semantic-geometric-decoupled-routing
IEEE
X. Long, W. Sun, W. Gao, P. Jiao, B. Qi, F. Zhu, Y. Zhong, S. Hoi, and B. Zhou, "Block-Sparse Attention with Semantic-Geometric Decoupled Routing," https://omanscience.com/ar/articles/block-sparse-attention-with-semantic-geometric-decoupled-routing.