Abstract
Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event. We find that joint multimodal diffusion transformers exhibit a corresponding asymmetry in cross-modal correspondence: companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. We express both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap. We introduce RecCAR, standing for Reciprocal Cross-modal Attention Regularization, a KL regularizer that uses the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it. Across joint video-motion and video-audio generation, RecCAR improves the Human Anatomy score from 0.69 to 0.75 and reduces audio-video desynchronization from 0.804 to 0.752, while improving overall generation
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Rahamim, O., Samuel, D., Schwartz, I., & Chechik, G. (2026). All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation. https://omanscience.com/en/articles/all-modalities-are-equal-but-video-is-more-equal-closing-the-cross-attention-gap-in-joint-video-generation
MLA 9
Rahamim, Ohad, et al. "All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation." https://omanscience.com/en/articles/all-modalities-are-equal-but-video-is-more-equal-closing-the-cross-attention-gap-in-joint-video-generation.
Chicago (author–date)
Rahamim, Ohad, Dvir Samuel, Idan Schwartz, and Gal Chechik. 2026. "All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation." https://omanscience.com/en/articles/all-modalities-are-equal-but-video-is-more-equal-closing-the-cross-attention-gap-in-joint-video-generation.
Harvard
Rahamim, O., Samuel, D., Schwartz, I. and Chechik, G. (2026) 'All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation', Available at: https://omanscience.com/en/articles/all-modalities-are-equal-but-video-is-more-equal-closing-the-cross-attention-gap-in-joint-video-generation.
Vancouver
Rahamim O, Samuel D, Schwartz I, Chechik G. All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation. https://omanscience.com/en/articles/all-modalities-are-equal-but-video-is-more-equal-closing-the-cross-attention-gap-in-joint-video-generation
IEEE
O. Rahamim, D. Samuel, I. Schwartz, and G. Chechik, "All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation," https://omanscience.com/en/articles/all-modalities-are-equal-but-video-is-more-equal-closing-the-cross-attention-gap-in-joint-video-generation.