Abstract
The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Solgi, R., Madinei, P., & Zhang, Z. (2026). Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling. https://omanscience.com/en/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling
MLA 9
Solgi, Ryan, et al. "Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling." https://omanscience.com/en/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.
Chicago (author–date)
Solgi, Ryan, Parsa Madinei, and Zheng Zhang. 2026. "Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling." https://omanscience.com/en/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.
Harvard
Solgi, R., Madinei, P. and Zhang, Z. (2026) 'Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling', Available at: https://omanscience.com/en/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.
Vancouver
Solgi R, Madinei P, Zhang Z. Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling. https://omanscience.com/en/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling
IEEE
R. Solgi, P. Madinei, and Z. Zhang, "Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling," https://omanscience.com/en/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.