Abstract
Recent works augment Vision-Language Models with geometry features from pretrained 3D models, expecting that the geometric signal will boost spatial reasoning. However, we find that simply fusing geometry features and training on standard spatial QA yields only marginal improvements on high-level multi-hop tasks. We attribute this gap to a training-signal problem: standard spatial QA can be largely answered from visual features and language priors, so the geometry pathway receives weak gradients and fails to integrate with the visual features. To provide a training signal that requires geometry, we propose \textbf{novel-view semantic rendering} as an auxiliary training task that requires the model to predict the semantic layout of an unobserved viewpoint, inspired by humans' ability to mentally simulate novel viewpoints during spatial reasoning. This task encourages joint use of both pathways: geometry provides pose-dependent visibility, while vision provides semantic content. Our auxiliary task yields consistent improvements over the geometry-augmented baseline across all three benchmarks (up to +1.6 on VSI-Bench, +2.2 on ReVSI, +2.9 on our 3D-Point-QA dataset) and our full model surpasses prior open-source methods on VSI-Bench and on ReVSI. Project page: https://yuqunw.github.io/Render2Reason/.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Wu, Y., Xiao, Y., Zou, C., Wang, S., & Hoiem, D. (2026). Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs. https://omanscience.com/en/articles/render-to-reason-novel-view-semantic-prediction-improves-spatial-understanding-in-vlms
MLA 9
Wu, Yuqun, et al. "Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs." https://omanscience.com/en/articles/render-to-reason-novel-view-semantic-prediction-improves-spatial-understanding-in-vlms.
Chicago (author–date)
Wu, Yuqun, Yao Xiao, Chuhang Zou, Shenlong Wang, and Derek Hoiem. 2026. "Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs." https://omanscience.com/en/articles/render-to-reason-novel-view-semantic-prediction-improves-spatial-understanding-in-vlms.
Harvard
Wu, Y., Xiao, Y., Zou, C., Wang, S. and Hoiem, D. (2026) 'Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs', Available at: https://omanscience.com/en/articles/render-to-reason-novel-view-semantic-prediction-improves-spatial-understanding-in-vlms.
Vancouver
Wu Y, Xiao Y, Zou C, Wang S, Hoiem D. Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs. https://omanscience.com/en/articles/render-to-reason-novel-view-semantic-prediction-improves-spatial-understanding-in-vlms
IEEE
Y. Wu, Y. Xiao, C. Zou, S. Wang, and D. Hoiem, "Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs," https://omanscience.com/en/articles/render-to-reason-novel-view-semantic-prediction-improves-spatial-understanding-in-vlms.