Abstract
Building a robotic manipulation system requires connecting perception, planning, and control through carefully designed representations and interfaces. VLM code generation offers a way to automate this construction, but independently generated components may operate on incompatible geometric and task-level information. We present Representation-guided Integration of VLM-generated Executable Task programs (RIVET), a framework for generating complete manipulation systems around a shared object-centric representation. The representation combines per-object 6D poses, which preserve the metric information required for action grounding, with a relation graph that exposes the task-level structure required for planning. Guided by this representation, a VLM generates cooperating perception, rendering, relation-inference, and planning programs, each combining task-specific computation with available packages where useful. The resulting programs are authored once for a manipulation domain and reused on unseen start and goal configurations without code regeneration. We evaluate RIVET on cube stacking, tangram rearrangement, and three-dimensional assembly in simulation and on a physical robot, where we achieve 83% overall success rate in the real world by reusing offline-generated systems. Our results demonstrate that representation-guided program generation can adapt a common manipulation framework to tasks with different geometric, relational, and sequential requirements.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yang, R., Yu, M., & Fan, C. (2026). Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation. https://omanscience.com/en/articles/representation-guided-generation-and-integration-of-executable-programs-for-robot-manipulation
MLA 9
Yang, Ruixiao, et al. "Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation." https://omanscience.com/en/articles/representation-guided-generation-and-integration-of-executable-programs-for-robot-manipulation.
Chicago (author–date)
Yang, Ruixiao, Mingxin Yu, and Chuchu Fan. 2026. "Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation." https://omanscience.com/en/articles/representation-guided-generation-and-integration-of-executable-programs-for-robot-manipulation.
Harvard
Yang, R., Yu, M. and Fan, C. (2026) 'Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation', Available at: https://omanscience.com/en/articles/representation-guided-generation-and-integration-of-executable-programs-for-robot-manipulation.
Vancouver
Yang R, Yu M, Fan C. Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation. https://omanscience.com/en/articles/representation-guided-generation-and-integration-of-executable-programs-for-robot-manipulation
IEEE
R. Yang, M. Yu, and C. Fan, "Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation," https://omanscience.com/en/articles/representation-guided-generation-and-integration-of-executable-programs-for-robot-manipulation.