Abstract
Large language models (LLMs) let users direct heterogeneous multi-robot systems (MRS) through natural language, but make task interpretation, robot assignment, and coordination difficult to inspect and change. Based on a formative study with 12 non-expert users, we developed MRPilot, a mixed reality system organized around four stages of supervision and intervention. MRPilot represents robot-team plans and execution states as structured commitments shared across synchronized situated and overview views. Across four stages, it helps users resolve ambiguous references (Forming), review plans before execution (Reviewing), monitor distributed execution (Following), and make robot-level or team-level changes when problems arise (Repairing). In a within-subjects study with 20 participants in a virtual reality-simulated home, MRPilot reduced workload, increased situational awareness, transparency, trust, and perceived control compared with a conventional LLM-based conversational interface using the same LLM planner and robot capabilities. We provide design implications for multi-scale intervention, adaptive supervision, and calibrated reliance in LLM-based MRS.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yang, X., Qian, X., Zhan, Y., Tran, N., Liu, Z., & Jin, Q. (2026). MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality. https://omanscience.com/en/articles/mrpilot-supervising-and-intervening-llm-based-multi-robot-teams-through-mixed-reality
MLA 9
Yang, Xiaoran, et al. "MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality." https://omanscience.com/en/articles/mrpilot-supervising-and-intervening-llm-based-multi-robot-teams-through-mixed-reality.
Chicago (author–date)
Yang, Xiaoran, Xun Qian, Yang Zhan, Nathan Tran, Ziyi Liu, and Qiao Jin. 2026. "MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality." https://omanscience.com/en/articles/mrpilot-supervising-and-intervening-llm-based-multi-robot-teams-through-mixed-reality.
Harvard
Yang, X., Qian, X., Zhan, Y., Tran, N., Liu, Z. and Jin, Q. (2026) 'MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality', Available at: https://omanscience.com/en/articles/mrpilot-supervising-and-intervening-llm-based-multi-robot-teams-through-mixed-reality.
Vancouver
Yang X, Qian X, Zhan Y, Tran N, Liu Z, Jin Q. MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality. https://omanscience.com/en/articles/mrpilot-supervising-and-intervening-llm-based-multi-robot-teams-through-mixed-reality
IEEE
X. Yang, X. Qian, Y. Zhan, N. Tran, Z. Liu, and Q. Jin, "MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality," https://omanscience.com/en/articles/mrpilot-supervising-and-intervening-llm-based-multi-robot-teams-through-mixed-reality.