Preprint Open access
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine different attention modules to improve long-context efficiency and performance in length extrapolation and context extension. To explain why hybrid models work and how to desi …