Abstract
Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder attention pathologies that suppress visual evidence and exacerbate hallucinations. In this paper, we revisit visual attention sinks and uncover a structured, layer-dependent behavior: across prompts, early and late decoder layers exhibit prompt-invariant attention collapse onto the same few image regions, which we term PIS (Prompt-Invariant Sinks), whereas mid layers become prompt-conditioned and drive vision-language alignment. This split suggests that treating sinks as a uniform effect is incomplete. Building on this insight, we propose SAGE (Sink-Aware Guided Emphasis), a lightweight intervention that steers decoder attention away from PIS and toward query-dependent regions of interest (ROIs) using token-aligned ROI masks derived from standard vision backbones such as CLIP, ViT, and DINOv3. Evaluated on diverse vision-encoder + decoder-only LLM VLM families, SAGE improves visual grounding, reduces hallucinations, and yields consistent gains across public downstream vision-language benchmarks, including fine-grained visual discrimination settings where localized evidence is crucial, when instantiated with backbone-derived ROI masks.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Song, J., & Yoo, Y. (2026). SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders. https://omanscience.com/en/articles/sage-sink-aware-guided-emphasis-for-visual-grounding-in-vision-language-decoders
MLA 9
Song, Jeonghyo, and YoungJoon Yoo. "SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders." https://omanscience.com/en/articles/sage-sink-aware-guided-emphasis-for-visual-grounding-in-vision-language-decoders.
Chicago (author–date)
Song, Jeonghyo, and YoungJoon Yoo. 2026. "SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders." https://omanscience.com/en/articles/sage-sink-aware-guided-emphasis-for-visual-grounding-in-vision-language-decoders.
Harvard
Song, J. and Yoo, Y. (2026) 'SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders', Available at: https://omanscience.com/en/articles/sage-sink-aware-guided-emphasis-for-visual-grounding-in-vision-language-decoders.
Vancouver
Song J, Yoo Y. SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders. https://omanscience.com/en/articles/sage-sink-aware-guided-emphasis-for-visual-grounding-in-vision-language-decoders
IEEE
J. Song, and Y. Yoo, "SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders," https://omanscience.com/en/articles/sage-sink-aware-guided-emphasis-for-visual-grounding-in-vision-language-decoders.