Abstract
Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on \texttt{balance-hard}, where Total Average Return (TAR) increases from $98\pm18$ to $387\pm255$, and \texttt{hurdle}, from $199\pm13$ to $466\pm200$; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Boyalakuntla, K., Liu, Y., & Boularias, A. (2026). Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models. https://omanscience.com/en/articles/beyond-policy-alignment-closing-the-planning-learning-loop-for-robot-control-with-learned-world-models
MLA 9
Boyalakuntla, Kowndinya, et al. "Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models." https://omanscience.com/en/articles/beyond-policy-alignment-closing-the-planning-learning-loop-for-robot-control-with-learned-world-models.
Chicago (author–date)
Boyalakuntla, Kowndinya, Yuhan Liu, and Abdeslam Boularias. 2026. "Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models." https://omanscience.com/en/articles/beyond-policy-alignment-closing-the-planning-learning-loop-for-robot-control-with-learned-world-models.
Harvard
Boyalakuntla, K., Liu, Y. and Boularias, A. (2026) 'Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models', Available at: https://omanscience.com/en/articles/beyond-policy-alignment-closing-the-planning-learning-loop-for-robot-control-with-learned-world-models.
Vancouver
Boyalakuntla K, Liu Y, Boularias A. Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models. https://omanscience.com/en/articles/beyond-policy-alignment-closing-the-planning-learning-loop-for-robot-control-with-learned-world-models
IEEE
K. Boyalakuntla, Y. Liu, and A. Boularias, "Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models," https://omanscience.com/en/articles/beyond-policy-alignment-closing-the-planning-learning-loop-for-robot-control-with-learned-world-models.