Abstract
Reinforcement learning is crucial for improving large language models' reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability. Asynchronous or partial-rollout methods improve throughput by relaxing synchronization, but inevitably introduce stale off-policy samples (trajectories) that may hurt final accuracy. Existing approaches mainly mitigate this off-policy issue by reweighting off-policy samples during training, yet they can still leave a performance gap compared to fully on-policy training. In this work, rather than passively reweighting samples during training, we propose RollVerify, a lightweight RL framework built on partial rollout that actively verifies and repairs samples before they enter training. Specifically, it introduces an off-policy shift metric OPS, to quantify the off-policy deviation of partially generated trajectories. Guided by the OPS constraint, RollVerify performs both sequence-level and token-level verification to identify and truncate invalid suffixes of trajectories. This yields high-quality samples that protect the models' accuracy while preserving the efficiency gains of partial rollout. Experiments on mathematical and tool-assisted mathematical reasoning show that RollVerify achieves accuracy comparable to on-policy training while reducing training cost. Additional code-generation results provide preliminary evidence beyond mathematics.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yao, Y., Tan, J., Liang, K., Yin, Z., Niu, Y., Gong, R., Lin, D., & Xu, N. (2026). RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning. https://omanscience.com/en/articles/rollverify-bridging-efficiency-and-accuracy-in-long-tail-rollout-reinforcement-learning
MLA 9
Yao, Yongqiang, et al. "RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning." https://omanscience.com/en/articles/rollverify-bridging-efficiency-and-accuracy-in-long-tail-rollout-reinforcement-learning.
Chicago (author–date)
Yao, Yongqiang, Jinru Tan, Kaihuan Liang, Zixin Yin, Yazhe Niu, Ruihao Gong, Dahua Lin, and Ningyi Xu. 2026. "RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning." https://omanscience.com/en/articles/rollverify-bridging-efficiency-and-accuracy-in-long-tail-rollout-reinforcement-learning.
Harvard
Yao, Y., Tan, J., Liang, K., Yin, Z., Niu, Y., Gong, R., Lin, D. and Xu, N. (2026) 'RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning', Available at: https://omanscience.com/en/articles/rollverify-bridging-efficiency-and-accuracy-in-long-tail-rollout-reinforcement-learning.
Vancouver
Yao Y, Tan J, Liang K, Yin Z, Niu Y, Gong R, et al. RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning. https://omanscience.com/en/articles/rollverify-bridging-efficiency-and-accuracy-in-long-tail-rollout-reinforcement-learning
IEEE
Y. Yao, J. Tan, K. Liang, Z. Yin, Y. Niu, R. Gong, D. Lin, and N. Xu, "RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning," https://omanscience.com/en/articles/rollverify-bridging-efficiency-and-accuracy-in-long-tail-rollout-reinforcement-learning.