Preprint Open access
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidd …