Preprint Open access
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-s …