Abstract
Object hallucination remains a major obstacle for large vision-language models (LVLMs) to generate reliable content. An intuitive mitigation strategy is to suppress hallucination-related components in hidden representations. However, these components may also contain useful information, and suppressing them can weaken the model's multimodal capabilities. In this paper, we propose ResOT, a training-free method that repairs representations at inference time through localized distribution alignment. Specifically, ResOT projects dominant hallucinated directions away from the faithful subspace, forming a low-dimensional residual subspace for intervention. Within this subspace, ResOT uses Gaussian optimal transport (OT) to align the hallucinated distribution with the faithful one. The resulting map defines repair targets with minimal changes to the original representations. At inference, ResOT adaptively controls how far each token state moves toward its OT target. Experiments on three representative LVLMs show that ResOT substantially reduces object hallucination while improving image caption quality and multimodal performance across multiple benchmarks. Code will be released.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhao, C., Dong, X., Shi, J., Peng, L., Wang, C., Lei, Z., He, R., & Du, B. (2026). From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment. https://omanscience.com/en/articles/from-suppression-to-repair-mitigating-object-hallucination-in-large-vision-language-models-via-localized-distribution-alignment
MLA 9
Zhao, Chen, et al. "From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment." https://omanscience.com/en/articles/from-suppression-to-repair-mitigating-object-hallucination-in-large-vision-language-models-via-localized-distribution-alignment.
Chicago (author–date)
Zhao, Chen, Xingping Dong, Jiachun Shi, Liang Peng, Chong Wang, Zhen Lei, Ran He, and Bo Du. 2026. "From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment." https://omanscience.com/en/articles/from-suppression-to-repair-mitigating-object-hallucination-in-large-vision-language-models-via-localized-distribution-alignment.
Harvard
Zhao, C., Dong, X., Shi, J., Peng, L., Wang, C., Lei, Z., He, R. and Du, B. (2026) 'From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment', Available at: https://omanscience.com/en/articles/from-suppression-to-repair-mitigating-object-hallucination-in-large-vision-language-models-via-localized-distribution-alignment.
Vancouver
Zhao C, Dong X, Shi J, Peng L, Wang C, Lei Z, et al. From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment. https://omanscience.com/en/articles/from-suppression-to-repair-mitigating-object-hallucination-in-large-vision-language-models-via-localized-distribution-alignment
IEEE
C. Zhao, X. Dong, J. Shi, L. Peng, C. Wang, Z. Lei, R. He, and B. Du, "From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment," https://omanscience.com/en/articles/from-suppression-to-repair-mitigating-object-hallucination-in-large-vision-language-models-via-localized-distribution-alignment.