Abstract
Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Robson, B., Mentu, S., Zhao, W., & Solin, A. (2026). LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning. https://omanscience.com/en/articles/leavjepa-a-minimalist-architecture-for-audio-visual-self-supervised-learning
MLA 9
Robson, Benjamin, et al. "LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning." https://omanscience.com/en/articles/leavjepa-a-minimalist-architecture-for-audio-visual-self-supervised-learning.
Chicago (author–date)
Robson, Benjamin, Santeri Mentu, Wenshuai Zhao, and Arno Solin. 2026. "LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning." https://omanscience.com/en/articles/leavjepa-a-minimalist-architecture-for-audio-visual-self-supervised-learning.
Harvard
Robson, B., Mentu, S., Zhao, W. and Solin, A. (2026) 'LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning', Available at: https://omanscience.com/en/articles/leavjepa-a-minimalist-architecture-for-audio-visual-self-supervised-learning.
Vancouver
Robson B, Mentu S, Zhao W, Solin A. LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning. https://omanscience.com/en/articles/leavjepa-a-minimalist-architecture-for-audio-visual-self-supervised-learning
IEEE
B. Robson, S. Mentu, W. Zhao, and A. Solin, "LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning," https://omanscience.com/en/articles/leavjepa-a-minimalist-architecture-for-audio-visual-self-supervised-learning.