Abstract
Multimodal large language models (MLLMs) have shown strong visual reasoning abilities, but knowledge-intensive visual question answering often requires external textual evidence beyond the image and the model's parametric knowledge. Existing multimodal RAG systems commonly rely on Top-$K$ retrieval or reranking, which may return redundant passages and provide limited control over whether an answer update is sufficiently supported by the retrieved evidence. We propose \textit{CLIMB}, a training-free inference-time framework for multimodal RAG. CLIMB first constructs a compact complementary evidence pool using an MMR-style objective that balances query relevance and passage-level redundancy. It then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when the estimated confidence increases. This design provides a simple stopping criterion and reduces unnecessary refinement without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show that CLIMB consistently improves over retrieval-augmented multimodal baselines. Ablations further indicate that complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute to the final performance.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Gao, H., Xu, W., Zhang, Z., Mei, K., Yang, J., & Metaxas, D. N. (2026). CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation. https://omanscience.com/en/articles/climb-confidence-guided-complementary-evidence-for-multimodal-retrieval-augmented-generation
MLA 9
Gao, Hang, et al. "CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation." https://omanscience.com/en/articles/climb-confidence-guided-complementary-evidence-for-multimodal-retrieval-augmented-generation.
Chicago (author–date)
Gao, Hang, Wujiang Xu, Zhixing Zhang, Kai Mei, Jingyi Yang, and Dimitris N. Metaxas. 2026. "CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation." https://omanscience.com/en/articles/climb-confidence-guided-complementary-evidence-for-multimodal-retrieval-augmented-generation.
Harvard
Gao, H., Xu, W., Zhang, Z., Mei, K., Yang, J. and Metaxas, D. N. (2026) 'CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation', Available at: https://omanscience.com/en/articles/climb-confidence-guided-complementary-evidence-for-multimodal-retrieval-augmented-generation.
Vancouver
Gao H, Xu W, Zhang Z, Mei K, Yang J, Metaxas DN. CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation. https://omanscience.com/en/articles/climb-confidence-guided-complementary-evidence-for-multimodal-retrieval-augmented-generation
IEEE
H. Gao, W. Xu, Z. Zhang, K. Mei, J. Yang, and D. N. Metaxas, "CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation," https://omanscience.com/en/articles/climb-confidence-guided-complementary-evidence-for-multimodal-retrieval-augmented-generation.