الملخص
Reinforcement learning (RL) methods such as GRPO substantially improve large language model reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating "less is more." This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at https://github.com/hexuandeng/GRPODropout/.
الكلمات المفتاحية
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Deng, H., Yan, Z., Liu, X., Nie, S., Wang, Y., Wang, C., Zhang, Z., Jiang, T., Xiao, Q., Zhang, J., & Zhang, M. (2026). GRPODropout: Less is More for Online Reinforcement Learning Rollouts. https://omanscience.com/ar/articles/grpodropout-less-is-more-for-online-reinforcement-learning-rollouts
MLA 9
Deng, Hexuan, et al. "GRPODropout: Less is More for Online Reinforcement Learning Rollouts." https://omanscience.com/ar/articles/grpodropout-less-is-more-for-online-reinforcement-learning-rollouts.
شيكاغو (المؤلف–التاريخ)
Deng, Hexuan, Zihao Yan, Xuebo Liu, Shuo Nie, Yue Wang, Chen Wang, Zhaohua Zhang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, and Min Zhang. 2026. "GRPODropout: Less is More for Online Reinforcement Learning Rollouts." https://omanscience.com/ar/articles/grpodropout-less-is-more-for-online-reinforcement-learning-rollouts.
هارفارد
Deng, H., Yan, Z., Liu, X., Nie, S., Wang, Y., Wang, C., Zhang, Z., Jiang, T., Xiao, Q., Zhang, J. and Zhang, M. (2026) 'GRPODropout: Less is More for Online Reinforcement Learning Rollouts', Available at: https://omanscience.com/ar/articles/grpodropout-less-is-more-for-online-reinforcement-learning-rollouts.
فانكوفر
Deng H, Yan Z, Liu X, Nie S, Wang Y, Wang C, et al. GRPODropout: Less is More for Online Reinforcement Learning Rollouts. https://omanscience.com/ar/articles/grpodropout-less-is-more-for-online-reinforcement-learning-rollouts
IEEE
H. Deng, Z. Yan, X. Liu, S. Nie, Y. Wang, C. Wang, Z. Zhang, T. Jiang, Q. Xiao, J. Zhang, and M. Zhang, "GRPODropout: Less is More for Online Reinforcement Learning Rollouts," https://omanscience.com/ar/articles/grpodropout-less-is-more-for-online-reinforcement-learning-rollouts.