Abstract
Large vision-language models (LVLMs) exhibit strong multimodal in-context learning (ICL) capabilities, yet this ability degrades substantially as model size decreases. Knowledge distillation offers a natural way to bridge this gap, but existing methods primarily align output distributions or hidden representations directly. Such alignment teaches the student what the teacher predicts without revealing which evidence in the complex context causally supports that prediction. Consequently, a student can imitate the teacher's answer while continuing to rely on language priors, prompt structure, or other spurious cues. To address this limitation, we introduce Multimodal Causal Distillation (MCD), a distillation framework that transfers how a strong teacher uses multimodal evidence during ICL. MCD uses structure-preserving token interventions to identify and verify causal evidence, then transfers how the teacher responds when that evidence is retained or removed. This design connects distillation to the causal patterns by which the model uses contextual evidence during multimodal ICL. Experiments across three LVLM families and seven benchmarks show that MCD improves student performance by 7.23 points on average and outperforms vanilla distillation by 4.68 points, while further analyses confirm the generalizability of these gains.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Li, Y., Li, J., Xiao, C., Xiao, X., Wang, T., & Liu, Y. (2026). MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models. https://omanscience.com/en/articles/mcd-causal-distillation-of-multimodal-in-context-learning-in-large-vision-language-models
MLA 9
Li, Yanshu, et al. "MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models." https://omanscience.com/en/articles/mcd-causal-distillation-of-multimodal-in-context-learning-in-large-vision-language-models.
Chicago (author–date)
Li, Yanshu, Jiaqian Li, Canran Xiao, Xi Xiao, Tianyang Wang, and Yongtai Liu. 2026. "MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models." https://omanscience.com/en/articles/mcd-causal-distillation-of-multimodal-in-context-learning-in-large-vision-language-models.
Harvard
Li, Y., Li, J., Xiao, C., Xiao, X., Wang, T. and Liu, Y. (2026) 'MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models', Available at: https://omanscience.com/en/articles/mcd-causal-distillation-of-multimodal-in-context-learning-in-large-vision-language-models.
Vancouver
Li Y, Li J, Xiao C, Xiao X, Wang T, Liu Y. MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models. https://omanscience.com/en/articles/mcd-causal-distillation-of-multimodal-in-context-learning-in-large-vision-language-models
IEEE
Y. Li, J. Li, C. Xiao, X. Xiao, T. Wang, and Y. Liu, "MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models," https://omanscience.com/en/articles/mcd-causal-distillation-of-multimodal-in-context-learning-in-large-vision-language-models.