Abstract
Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Wang, Z., Zhang, S., Zhang, Y., Zhang, H., Xu, J., Xu, B., Hu, J., Zou, Y., Sapra, K., Tao, A., Kautz, J., & Dong, Y. (2026). OSWorld-Pro: Process-based Evaluation for Computer Use Agents. https://omanscience.com/en/articles/osworld-pro-process-based-evaluation-for-computer-use-agents
MLA 9
Wang, Zhilin, et al. "OSWorld-Pro: Process-based Evaluation for Computer Use Agents." https://omanscience.com/en/articles/osworld-pro-process-based-evaluation-for-computer-use-agents.
Chicago (author–date)
Wang, Zhilin, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, Karan Sapra, Andrew Tao, Jan Kautz, and Yi Dong. 2026. "OSWorld-Pro: Process-based Evaluation for Computer Use Agents." https://omanscience.com/en/articles/osworld-pro-process-based-evaluation-for-computer-use-agents.
Harvard
Wang, Z., Zhang, S., Zhang, Y., Zhang, H., Xu, J., Xu, B., Hu, J., Zou, Y., Sapra, K., Tao, A., Kautz, J. and Dong, Y. (2026) 'OSWorld-Pro: Process-based Evaluation for Computer Use Agents', Available at: https://omanscience.com/en/articles/osworld-pro-process-based-evaluation-for-computer-use-agents.
Vancouver
Wang Z, Zhang S, Zhang Y, Zhang H, Xu J, Xu B, et al. OSWorld-Pro: Process-based Evaluation for Computer Use Agents. https://omanscience.com/en/articles/osworld-pro-process-based-evaluation-for-computer-use-agents
IEEE
Z. Wang, S. Zhang, Y. Zhang, H. Zhang, J. Xu, B. Xu, J. Hu, Y. Zou, K. Sapra, A. Tao, J. Kautz, and Y. Dong, "OSWorld-Pro: Process-based Evaluation for Computer Use Agents," https://omanscience.com/en/articles/osworld-pro-process-based-evaluation-for-computer-use-agents.