Abstract
Many works approach continual learning through the lens of infinite context windows. As an agent puts more observation into context (concretely the KV cache), compacting said context is akin to direct memory manipulation, without affecting the base model's weights. Many works pose KV compaction as an optimization problem: learn a smaller set of KV vectors that matches the behavior of the full KV cache. While this preserves base model behavior, optimizing through a frozen base model results in a highly nontrivial optimization problem with a brittle and flat loss landscape. In this paper, we characterize what makes these optimization problems difficult and demonstrate that a heavily simplified Perceiver-based architecture not only matches performance of a full Perceiver transformer in continuous context compaction, but outperforms baselines on compaction utility. Results are presented on MCQ tasks across Finance, Legal, Gutenberg, and Code.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Villeneuve, T., Sandomirsky, A., O'Neill, C., Kirkby, M., & Psenka, M. (2026). The Optimization Landscape of Learning Compacted Context Models. https://omanscience.com/en/articles/the-optimization-landscape-of-learning-compacted-context-models
MLA 9
Villeneuve, Thomas, et al. "The Optimization Landscape of Learning Compacted Context Models." https://omanscience.com/en/articles/the-optimization-landscape-of-learning-compacted-context-models.
Chicago (author–date)
Villeneuve, Thomas, Alex Sandomirsky, Charles O'Neill, Max Kirkby, and Michael Psenka. 2026. "The Optimization Landscape of Learning Compacted Context Models." https://omanscience.com/en/articles/the-optimization-landscape-of-learning-compacted-context-models.
Harvard
Villeneuve, T., Sandomirsky, A., O'Neill, C., Kirkby, M. and Psenka, M. (2026) 'The Optimization Landscape of Learning Compacted Context Models', Available at: https://omanscience.com/en/articles/the-optimization-landscape-of-learning-compacted-context-models.
Vancouver
Villeneuve T, Sandomirsky A, O'Neill C, Kirkby M, Psenka M. The Optimization Landscape of Learning Compacted Context Models. https://omanscience.com/en/articles/the-optimization-landscape-of-learning-compacted-context-models
IEEE
T. Villeneuve, A. Sandomirsky, C. O'Neill, M. Kirkby, and M. Psenka, "The Optimization Landscape of Learning Compacted Context Models," https://omanscience.com/en/articles/the-optimization-landscape-of-learning-compacted-context-models.