Abstract
Audio-visual large language models (AVLLMs) have made remarkable progress in multimodal understanding and reasoning through interactions among visual, auditory, and linguistic information. However, recent studies show that AVLLMs face a critical challenge: $\textbf{source-confused grounding hallucination}$, where cues from the unused modality induce responses that the required modality does not support, undermining reliability in real-world applications. Existing methods have made progress in mitigating this failure, yet how it arises from internal cross-modal interactions remains insufficiently understood. To address this gap, we conduct path-intervention and representation analyses, revealing a $\textbf{question-relay}$ mechanism: question states carry interfering cues alongside required-source evidence, undermining grounding in required-modality evidence. Cutting pathways from interfering modality to question states yields greater correct-answer logit recovery than cutting those to the generation position. Motivated by these findings, we propose $\textbf{SECRET}$ ($\textbf{S}$ourc$\textbf{E}$-$\textbf{C}$onditioned $\textbf{RE}$lay s$\textbf{T}$eering), a training-free method that mitigates cross-modal interference at the question relay. Using contrasting question representations elicited through different modality-pathway interventions, SECRET steers the original question states toward required-source evidence. Experiments on two widely adopted benchmarks CMM and AVHBench across three AVLLMs show that SECRET consistently outperforms prior training-free methods, substantially mitigating source-confused grounding hallucinations (e.g., up to +18.0 and +7.1 percentage points over base models). Modality-specific captioning further demonstrates its generalizability to open-ended generation.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhang, Y., Zhang, P., Bai, X., Zhang, P., Xiang, Y., & Chen, K. (2026). Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models. https://omanscience.com/en/articles/devils-in-question-relay-source-conditioned-relay-steering-to-mitigate-hallucinations-in-audio-visual-large-language-models
MLA 9
Zhang, Yu, et al. "Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models." https://omanscience.com/en/articles/devils-in-question-relay-source-conditioned-relay-steering-to-mitigate-hallucinations-in-audio-visual-large-language-models.
Chicago (author–date)
Zhang, Yu, Pingrui Zhang, Xuefeng Bai, Pengfei Zhang, Yang Xiang, and Kehai Chen. 2026. "Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models." https://omanscience.com/en/articles/devils-in-question-relay-source-conditioned-relay-steering-to-mitigate-hallucinations-in-audio-visual-large-language-models.
Harvard
Zhang, Y., Zhang, P., Bai, X., Zhang, P., Xiang, Y. and Chen, K. (2026) 'Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models', Available at: https://omanscience.com/en/articles/devils-in-question-relay-source-conditioned-relay-steering-to-mitigate-hallucinations-in-audio-visual-large-language-models.
Vancouver
Zhang Y, Zhang P, Bai X, Zhang P, Xiang Y, Chen K. Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models. https://omanscience.com/en/articles/devils-in-question-relay-source-conditioned-relay-steering-to-mitigate-hallucinations-in-audio-visual-large-language-models
IEEE
Y. Zhang, P. Zhang, X. Bai, P. Zhang, Y. Xiang, and K. Chen, "Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models," https://omanscience.com/en/articles/devils-in-question-relay-source-conditioned-relay-steering-to-mitigate-hallucinations-in-audio-visual-large-language-models.