Abstract
Gated Linear Attention (GLA) Transformers advance linear recurrent models through data-dependent gating, but face a core limitation: the fixed-capacity memory matrices across all heads operate at a single temporal resolution, where each token is processed individually, forcing them to simultaneously encode local syntactic patterns and long-range semantic structure, creating a representational bottleneck that gating alone is insufficient to resolve. We introduce Multi-Scale Gated Linear Attention (MS-GLA), which addresses this by distributing attention heads across multiple temporal resolutions. Coarser resolutions pool longer token spans naturally specializing toward long-range dependencies, while finer head groups retain sensitivity to local syntactic structure. A learnable, input-dependent fusion layer dynamically recombines head group outputs at each timestep, expanding effective memory capacity without increasing per-head state size. This multi-resolution decomposition draws on principles from Multi-Scale State-Space Models (MS-SSM), adapting them to the gated linear attention setting. We evaluate MS-GLA on language modeling, recall-intensive tasks, and long-context generalization. Across all settings, MS-GLA consistently achieves higher accuracy and lower perplexity than GLA at matched parameter counts, with up to 18.9% improvement on recall-intensive tasks and 9.5% lower average perplexity on language modeling benchmarks, validating multi-temporal resolution decomposition as a principled and effective extension of Gated Linear Attention.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Dev, P., Sankar, A., & Varma, V. (2026). MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution. https://omanscience.com/en/articles/ms-gla-multi-scale-gated-linear-attention-for-addressing-representational-bottlenecks-via-multi-temporal-resolution
MLA 9
Dev, Prasoon, et al. "MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution." https://omanscience.com/en/articles/ms-gla-multi-scale-gated-linear-attention-for-addressing-representational-bottlenecks-via-multi-temporal-resolution.
Chicago (author–date)
Dev, Prasoon, Anirudh Sankar, and Vasudeva Varma. 2026. "MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution." https://omanscience.com/en/articles/ms-gla-multi-scale-gated-linear-attention-for-addressing-representational-bottlenecks-via-multi-temporal-resolution.
Harvard
Dev, P., Sankar, A. and Varma, V. (2026) 'MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution', Available at: https://omanscience.com/en/articles/ms-gla-multi-scale-gated-linear-attention-for-addressing-representational-bottlenecks-via-multi-temporal-resolution.
Vancouver
Dev P, Sankar A, Varma V. MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution. https://omanscience.com/en/articles/ms-gla-multi-scale-gated-linear-attention-for-addressing-representational-bottlenecks-via-multi-temporal-resolution
IEEE
P. Dev, A. Sankar, and V. Varma, "MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution," https://omanscience.com/en/articles/ms-gla-multi-scale-gated-linear-attention-for-addressing-representational-bottlenecks-via-multi-temporal-resolution.