نسخة أولية وصول مفتوح
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context p …