الملخص
The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Solgi, R., Madinei, P., & Zhang, Z. (2026). Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling. https://omanscience.com/ar/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling
MLA 9
Solgi, Ryan, et al. "Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling." https://omanscience.com/ar/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.
شيكاغو (المؤلف–التاريخ)
Solgi, Ryan, Parsa Madinei, and Zheng Zhang. 2026. "Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling." https://omanscience.com/ar/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.
هارفارد
Solgi, R., Madinei, P. and Zhang, Z. (2026) 'Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling', Available at: https://omanscience.com/ar/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.
فانكوفر
Solgi R, Madinei P, Zhang Z. Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling. https://omanscience.com/ar/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling
IEEE
R. Solgi, P. Madinei, and Z. Zhang, "Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling," https://omanscience.com/ar/articles/tucker-bottleneck-attention-for-multi-dimensional-sequence-modeling.