Abstract
Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Wang, Y., Yao, J., Liu, L., Lu, P., & Xu, Z. Q. J. (2026). Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks. https://omanscience.com/en/articles/probability-signature-dynamics-unpacking-modular-addition-learning-within-two-layer-networks
MLA 9
Wang, Yunji, et al. "Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks." https://omanscience.com/en/articles/probability-signature-dynamics-unpacking-modular-addition-learning-within-two-layer-networks.
Chicago (author–date)
Wang, Yunji, Junjie Yao, Linyu Liu, Pinyan Lu, and Zhi-Qin John Xu. 2026. "Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks." https://omanscience.com/en/articles/probability-signature-dynamics-unpacking-modular-addition-learning-within-two-layer-networks.
Harvard
Wang, Y., Yao, J., Liu, L., Lu, P. and Xu, Z. Q. J. (2026) 'Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks', Available at: https://omanscience.com/en/articles/probability-signature-dynamics-unpacking-modular-addition-learning-within-two-layer-networks.
Vancouver
Wang Y, Yao J, Liu L, Lu P, Xu ZQJ. Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks. https://omanscience.com/en/articles/probability-signature-dynamics-unpacking-modular-addition-learning-within-two-layer-networks
IEEE
Y. Wang, J. Yao, L. Liu, P. Lu, and Z. Q. J. Xu, "Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks," https://omanscience.com/en/articles/probability-signature-dynamics-unpacking-modular-addition-learning-within-two-layer-networks.