Abstract
World Action Models (WAMs) couple visual dynamics prediction with action generation, yet they do not explicitly support the reuse of action experience across manipulation tasks. Furthermore, existing WAMs struggle to capture underlying cross-task semantic relationships that could guide target action prediction, as redundant background elements interfere with the extraction of key visual information. To address these challenges, we develop a novel Action Experience Dictionary (AED) that encodes historical physical action trajectories into shared action embeddings to support skill reuse and model cross-task relationships. Specifically, we first aggregate historical actions to align with visual observations and retrieve action embeddings from the AED using a pretrained action tokenizer. Subsequently, we visually condition the pooled embeddings through cross-attention and prepend them to noisy action tokens, providing interaction context and action intent for prediction. To model action-related motion and reduce reliance on irrelevant background cues, we introduce a motion-aware transition loss that supervises visual feature change prediction over random temporal intervals. Experiments on simulation benchmarks and in real-world cross-embodiment settings verify the effectiveness of our AED. The project code is available at https://github.com/JiahuaDong/AED .
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Lyu, Q., Dong, J., Shen, H., Wang, X., Yu, H., Liu, B., Ding, H., Han, Z., Sebe, N., Laptev, I., Khan, F. S., & Khan, S. (2026). Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models. https://omanscience.com/en/articles/learning-skills-from-historical-action-trajectories-action-experience-dictionary-for-world-action-models
MLA 9
Lyu, Qi, et al. "Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models." https://omanscience.com/en/articles/learning-skills-from-historical-action-trajectories-action-experience-dictionary-for-world-action-models.
Chicago (author–date)
Lyu, Qi, Jiahua Dong, Hao Shen, Xudong Wang, Hongyuan Yu, Baichen Liu, Henghui Ding, Zhi Han, Nicu Sebe, Ivan Laptev, Fahad Shahbaz Khan, and Salman Khan. 2026. "Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models." https://omanscience.com/en/articles/learning-skills-from-historical-action-trajectories-action-experience-dictionary-for-world-action-models.
Harvard
Lyu, Q., Dong, J., Shen, H., Wang, X., Yu, H., Liu, B., Ding, H., Han, Z., Sebe, N., Laptev, I., Khan, F. S. and Khan, S. (2026) 'Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models', Available at: https://omanscience.com/en/articles/learning-skills-from-historical-action-trajectories-action-experience-dictionary-for-world-action-models.
Vancouver
Lyu Q, Dong J, Shen H, Wang X, Yu H, Liu B, et al. Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models. https://omanscience.com/en/articles/learning-skills-from-historical-action-trajectories-action-experience-dictionary-for-world-action-models
IEEE
Q. Lyu, J. Dong, H. Shen, X. Wang, H. Yu, B. Liu, H. Ding, Z. Han, N. Sebe, I. Laptev, F. S. Khan, and S. Khan, "Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models," https://omanscience.com/en/articles/learning-skills-from-historical-action-trajectories-action-experience-dictionary-for-world-action-models.