Abstract
We propose a method for choosing the shared memory parameter $β_1=β_2=β$ in Adam from a short pilot training. The selected $β$ remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared $β=0.95$. The mean gap is also 32.3% lower than that of the best constant $β$ chosen across all eleven workloads.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Fernández-Hernández, A., Pérez-Corral, C., Mestre, J. I., Dolz, M. F., & Quintana-Ortí, E. S. (2026). Early Memory Selection for Balanced Adam. https://omanscience.com/en/articles/early-memory-selection-for-balanced-adam
MLA 9
Fernández-Hernández, Alberto, et al. "Early Memory Selection for Balanced Adam." https://omanscience.com/en/articles/early-memory-selection-for-balanced-adam.
Chicago (author–date)
Fernández-Hernández, Alberto, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, and Enrique S. Quintana-Ortí. 2026. "Early Memory Selection for Balanced Adam." https://omanscience.com/en/articles/early-memory-selection-for-balanced-adam.
Harvard
Fernández-Hernández, A., Pérez-Corral, C., Mestre, J. I., Dolz, M. F. and Quintana-Ortí, E. S. (2026) 'Early Memory Selection for Balanced Adam', Available at: https://omanscience.com/en/articles/early-memory-selection-for-balanced-adam.
Vancouver
Fernández-Hernández A, Pérez-Corral C, Mestre JI, Dolz MF, Quintana-Ortí ES. Early Memory Selection for Balanced Adam. https://omanscience.com/en/articles/early-memory-selection-for-balanced-adam
IEEE
A. Fernández-Hernández, C. Pérez-Corral, J. I. Mestre, M. F. Dolz, and E. S. Quintana-Ortí, "Early Memory Selection for Balanced Adam," https://omanscience.com/en/articles/early-memory-selection-for-balanced-adam.