Abstract
Optimizer momentum is usually stored as a parameter-sized moving average of past gradients, which makes history costly and fixes each past signal in the coordinates in which it was computed. We introduce Backpropagated Output Momentum (BOM), which instead stores a compact moving average of prediction errors at the model output and reprojects that history through the current network at every step. A batch-level analysis characterizes the information retained and omitted by this relocation, while the implementation preserves the current supervised gradient and can replace the first-moment component of several adaptive optimizers. As a plug-in for momentum-based optimizers, including ones that already compress their state, BOM reduces parameter-shaped optimizer state by 49.7-99.8% in three compositions and, averaged over three language backbones, paired step time by 4.0%. It also improves mean validation performance across language and vision fine-tuning, by 1.42 points in the primary five-task comparison. Language and vision pretraining studies, together with matched mechanism controls, further test the construction across output spaces and model scales.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Li, Y., Fan, Z., Tran, N. H., & Yong, K. T. (2026). Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space. https://omanscience.com/en/articles/backpropagated-output-momentum-relocating-optimizer-history-from-parameters-to-task-space
MLA 9
Li, Yuchen, et al. "Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space." https://omanscience.com/en/articles/backpropagated-output-momentum-relocating-optimizer-history-from-parameters-to-task-space.
Chicago (author–date)
Li, Yuchen, Zongqi Fan, Nguyen H. Tran, and Ken-Tye Yong. 2026. "Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space." https://omanscience.com/en/articles/backpropagated-output-momentum-relocating-optimizer-history-from-parameters-to-task-space.
Harvard
Li, Y., Fan, Z., Tran, N. H. and Yong, K. T. (2026) 'Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space', Available at: https://omanscience.com/en/articles/backpropagated-output-momentum-relocating-optimizer-history-from-parameters-to-task-space.
Vancouver
Li Y, Fan Z, Tran NH, Yong KT. Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space. https://omanscience.com/en/articles/backpropagated-output-momentum-relocating-optimizer-history-from-parameters-to-task-space
IEEE
Y. Li, Z. Fan, N. H. Tran, and K. T. Yong, "Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space," https://omanscience.com/en/articles/backpropagated-output-momentum-relocating-optimizer-history-from-parameters-to-task-space.