Abstract
Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cache, so memory still grows with compute. Inference-time techniques can shrink this cache at a cost in quality. We pretrain looped language models to share memory: only the first recursion writes a cache, and later recursions read it while keeping a short window of their own. Surprisingly, we find that sharing memory does not cost quality and instead improves it. At 150M-1B parameters, our Looped Prediction Transformer (LPT) and its hybrid variant set a new quality-memory frontier for looped models: with five recursions, the hybrid lowers validation perplexity on FineWeb-Edu by 1.12-1.82 relative to a same-size standard Transformer while using 76-79% less context memory. Through an extensive analysis, we investigate why memory sharing helps. Shared and local memory develop different representations, and later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Monea, G., Ramji, K., El-Kurdi, Y., Lastras, L. A., Artzi, Y., Godey, N., & Astudillo, R. F. (2026). The Surprising Effectiveness of Shared Memory in Looped Transformers. https://omanscience.com/en/articles/the-surprising-effectiveness-of-shared-memory-in-looped-transformers
MLA 9
Monea, Giovanni, et al. "The Surprising Effectiveness of Shared Memory in Looped Transformers." https://omanscience.com/en/articles/the-surprising-effectiveness-of-shared-memory-in-looped-transformers.
Chicago (author–date)
Monea, Giovanni, Keshav Ramji, Yousef El-Kurdi, Luis A. Lastras, Yoav Artzi, Nathan Godey, and Ramón Fernandez Astudillo. 2026. "The Surprising Effectiveness of Shared Memory in Looped Transformers." https://omanscience.com/en/articles/the-surprising-effectiveness-of-shared-memory-in-looped-transformers.
Harvard
Monea, G., Ramji, K., El-Kurdi, Y., Lastras, L. A., Artzi, Y., Godey, N. and Astudillo, R. F. (2026) 'The Surprising Effectiveness of Shared Memory in Looped Transformers', Available at: https://omanscience.com/en/articles/the-surprising-effectiveness-of-shared-memory-in-looped-transformers.
Vancouver
Monea G, Ramji K, El-Kurdi Y, Lastras LA, Artzi Y, Godey N, et al. The Surprising Effectiveness of Shared Memory in Looped Transformers. https://omanscience.com/en/articles/the-surprising-effectiveness-of-shared-memory-in-looped-transformers
IEEE
G. Monea, K. Ramji, Y. El-Kurdi, L. A. Lastras, Y. Artzi, N. Godey, and R. F. Astudillo, "The Surprising Effectiveness of Shared Memory in Looped Transformers," https://omanscience.com/en/articles/the-surprising-effectiveness-of-shared-memory-in-looped-transformers.