Abstract
Monocular videos of human manipulation provide abundant dexterous demonstrations, yet reconstructing hand-object interaction from a single view and transferring it to robot hands remain difficult, limiting their direct use for robot execution. Prior methods either require task-specific RL training, limiting scalability, or assume clean motion-capture trajectories and thus cannot operate directly on video. We present OmniHOI, a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands. The key idea is to enforce physical consistency using the evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-in-the-loop refinement. Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy. Across 150 motion-capture trajectories transferred to each of five dexterous hands with 6 to 22 DoF, we achieve 39-89% success, compared with at most 31% for prior transfer methods. On 60 monocular video clips, we achieve 53% success, compared with 28% for the best prior video-to-robot pipeline. Its trajectories also execute on a real bimanual robot across diverse tasks.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Mao, T., Shao, Y., Wang, Z., Liu, H., Wang, Y., Wu, X., & Mu, Y. (2026). OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video. https://omanscience.com/en/articles/omnihoi-dexterous-hand-object-interaction-from-monocular-human-video
MLA 9
Mao, Ting, et al. "OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video." https://omanscience.com/en/articles/omnihoi-dexterous-hand-object-interaction-from-monocular-human-video.
Chicago (author–date)
Mao, Ting, Yanming Shao, Ziheng Wang, Haoyu Liu, Yiqun Wang, Xuanye Wu, and Yao Mu. 2026. "OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video." https://omanscience.com/en/articles/omnihoi-dexterous-hand-object-interaction-from-monocular-human-video.
Harvard
Mao, T., Shao, Y., Wang, Z., Liu, H., Wang, Y., Wu, X. and Mu, Y. (2026) 'OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video', Available at: https://omanscience.com/en/articles/omnihoi-dexterous-hand-object-interaction-from-monocular-human-video.
Vancouver
Mao T, Shao Y, Wang Z, Liu H, Wang Y, Wu X, et al. OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video. https://omanscience.com/en/articles/omnihoi-dexterous-hand-object-interaction-from-monocular-human-video
IEEE
T. Mao, Y. Shao, Z. Wang, H. Liu, Y. Wang, X. Wu, and Y. Mu, "OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video," https://omanscience.com/en/articles/omnihoi-dexterous-hand-object-interaction-from-monocular-human-video.