Abstract
Lightweight scene proxies let creators control scene layout and motion while leaving room for imagination in appearance, lighting, and visual effects. However, a suitable proxy is not uniquely defined, making paired proxy-video data difficult to construct automatically at scale. We present Proxy2World, a controllable world model that learns these complementary capabilities from ordinary posed RGBD videos, without training on authored proxy-video pairs. The model jointly learns depth-conditioned RGB generation and joint RGBD generation through cross-modal flow matching. Learning both tasks enables proxy-camera hybrid denoising at inference to follow the proxy structure while producing natural, detailed visuals. We further introduce ProxyBench to evaluate this capability across a diverse set of scenes, camera trajectories, and subject motions. Experiments on ProxyBench show that Proxy2World achieves a better balance between structural adherence and visual quality than camera-controlled and geometry-conditioned methods, supported by quantitative metrics, VLM assessments, human evaluations and diverse qualitative results.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Xu, H., Yan, W., Wang, A., Zou, C., Hong, S., & Huang, J. (2026). Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them. https://omanscience.com/en/articles/proxy2world-learning-to-generate-worlds-from-lightweight-proxies-without-seeing-them
MLA 9
Xu, Hongli, et al. "Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them." https://omanscience.com/en/articles/proxy2world-learning-to-generate-worlds-from-lightweight-proxies-without-seeing-them.
Chicago (author–date)
Xu, Hongli, Weilong Yan, Anbang Wang, Chunyu Zou, Siyu Hong, and Jingwei Huang. 2026. "Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them." https://omanscience.com/en/articles/proxy2world-learning-to-generate-worlds-from-lightweight-proxies-without-seeing-them.
Harvard
Xu, H., Yan, W., Wang, A., Zou, C., Hong, S. and Huang, J. (2026) 'Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them', Available at: https://omanscience.com/en/articles/proxy2world-learning-to-generate-worlds-from-lightweight-proxies-without-seeing-them.
Vancouver
Xu H, Yan W, Wang A, Zou C, Hong S, Huang J. Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them. https://omanscience.com/en/articles/proxy2world-learning-to-generate-worlds-from-lightweight-proxies-without-seeing-them
IEEE
H. Xu, W. Yan, A. Wang, C. Zou, S. Hong, and J. Huang, "Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them," https://omanscience.com/en/articles/proxy2world-learning-to-generate-worlds-from-lightweight-proxies-without-seeing-them.