Abstract
An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhu, K., Ye, X., Li, Y., Qian, C., Li, B., & Ji, H. (2026). Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training. https://omanscience.com/en/articles/agent-error-dataset-scaling-50-000-error-diagnosis-pairs-for-failure-analysis-and-error-aware-post-training
MLA 9
Zhu, Kunlun, et al. "Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training." https://omanscience.com/en/articles/agent-error-dataset-scaling-50-000-error-diagnosis-pairs-for-failure-analysis-and-error-aware-post-training.
Chicago (author–date)
Zhu, Kunlun, Xuyan Ye, Yibo Li, Cheng Qian, Beibin Li, and Heng Ji. 2026. "Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training." https://omanscience.com/en/articles/agent-error-dataset-scaling-50-000-error-diagnosis-pairs-for-failure-analysis-and-error-aware-post-training.
Harvard
Zhu, K., Ye, X., Li, Y., Qian, C., Li, B. and Ji, H. (2026) 'Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training', Available at: https://omanscience.com/en/articles/agent-error-dataset-scaling-50-000-error-diagnosis-pairs-for-failure-analysis-and-error-aware-post-training.
Vancouver
Zhu K, Ye X, Li Y, Qian C, Li B, Ji H. Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training. https://omanscience.com/en/articles/agent-error-dataset-scaling-50-000-error-diagnosis-pairs-for-failure-analysis-and-error-aware-post-training
IEEE
K. Zhu, X. Ye, Y. Li, C. Qian, B. Li, and H. Ji, "Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training," https://omanscience.com/en/articles/agent-error-dataset-scaling-50-000-error-diagnosis-pairs-for-failure-analysis-and-error-aware-post-training.