Abstract
Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify inference and dictionary learning as natural-gradient flows on a shared variational free energy. We instantiate this framework as BeFOND, an encoder-free sparse coding model with closed-form inference and learning dynamics. We show how recurrent explaining away reduces interference between overlapping features, while Fisher preconditioning can compensate for the slow learning of rare features. On synthetic data, BeFOND improves dictionary recovery and rare-feature detection, with a growing advantage over amortized baselines as superposition increases. On language-model activations, it improves single-feature concept detection and selective intervention, outperforming pretrained reference SAEs with substantially less training data. Its feature quality continues to improve with dictionary width, whereas the evaluated baselines largely plateau. Together, these results show how improving inference and learning within a unified probabilistic framework can make better use of data and dictionary capacity to interpret and intervene on neural representations.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Vafaii, H., Rao, T., Chanin, D., Fel, T., Yates, J. L., Olshausen, B., Klindt, D., George, D., & Lázaro-Gredilla, M. (2026). Inference and learning in sparse autoencoders as natural gradient flow. https://omanscience.com/en/articles/inference-and-learning-in-sparse-autoencoders-as-natural-gradient-flow
MLA 9
Vafaii, Hadi, et al. "Inference and learning in sparse autoencoders as natural gradient flow." https://omanscience.com/en/articles/inference-and-learning-in-sparse-autoencoders-as-natural-gradient-flow.
Chicago (author–date)
Vafaii, Hadi, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates, Bruno Olshausen, David Klindt, Dileep George, and Miguel Lázaro-Gredilla. 2026. "Inference and learning in sparse autoencoders as natural gradient flow." https://omanscience.com/en/articles/inference-and-learning-in-sparse-autoencoders-as-natural-gradient-flow.
Harvard
Vafaii, H., Rao, T., Chanin, D., Fel, T., Yates, J. L., Olshausen, B., Klindt, D., George, D. and Lázaro-Gredilla, M. (2026) 'Inference and learning in sparse autoencoders as natural gradient flow', Available at: https://omanscience.com/en/articles/inference-and-learning-in-sparse-autoencoders-as-natural-gradient-flow.
Vancouver
Vafaii H, Rao T, Chanin D, Fel T, Yates JL, Olshausen B, et al. Inference and learning in sparse autoencoders as natural gradient flow. https://omanscience.com/en/articles/inference-and-learning-in-sparse-autoencoders-as-natural-gradient-flow
IEEE
H. Vafaii, T. Rao, D. Chanin, T. Fel, J. L. Yates, B. Olshausen, D. Klindt, D. George, and M. Lázaro-Gredilla, "Inference and learning in sparse autoencoders as natural gradient flow," https://omanscience.com/en/articles/inference-and-learning-in-sparse-autoencoders-as-natural-gradient-flow.