Abstract
Long-horizon LLM agents are typically trained with sparse outcome rewards, making trajectory-level objectives too coarse to distinguish the contribution of individual decisions. Step-level credit assignment provides finer-grained supervision, but its estimates can be unreliable because observed returns also depend on subsequent actions, environment transitions, and trajectory length. We propose AdaStep, an Adaptive Step-credit weighting method that controls how strongly each group-derived local advantage modifies the trajectory-level signal. We formulate this weighting as a mean-squared-error estimation problem for the latent step advantage and, under an explicit conditional sampling assumption, derive an optimal per-state shrinkage coefficient. The coefficient admits a signal-to-total-variance interpretation: it preserves local credit when return variation is attributable to the selected action and suppresses it when variation is dominated by downstream randomness. AdaStep requires only lightweight scalar computation, with no critic, additional rollouts, or extra model inference. Experiments with three model backbones on ALFWorld, WebShop, and ScienceWorld show consistent improvements over baselines at low computational cost.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Wang, X., Wu, W., Zhang, M., Wang, Z., Shao, K., & Luan, J. (2026). AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning. https://omanscience.com/en/articles/adastep-adaptive-step-credit-weighting-for-agentic-reinforcement-learning
MLA 9
Wang, Xin, et al. "AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning." https://omanscience.com/en/articles/adastep-adaptive-step-credit-weighting-for-agentic-reinforcement-learning.
Chicago (author–date)
Wang, Xin, Wenhao Wu, Menghao Zhang, Zhi Wang, Kun Shao, and Jian Luan. 2026. "AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning." https://omanscience.com/en/articles/adastep-adaptive-step-credit-weighting-for-agentic-reinforcement-learning.
Harvard
Wang, X., Wu, W., Zhang, M., Wang, Z., Shao, K. and Luan, J. (2026) 'AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning', Available at: https://omanscience.com/en/articles/adastep-adaptive-step-credit-weighting-for-agentic-reinforcement-learning.
Vancouver
Wang X, Wu W, Zhang M, Wang Z, Shao K, Luan J. AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning. https://omanscience.com/en/articles/adastep-adaptive-step-credit-weighting-for-agentic-reinforcement-learning
IEEE
X. Wang, W. Wu, M. Zhang, Z. Wang, K. Shao, and J. Luan, "AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning," https://omanscience.com/en/articles/adastep-adaptive-step-credit-weighting-for-agentic-reinforcement-learning.