Abstract
Open-ended generation lacks canonical answers, making pointwise rewards difficult to calibrate for group-based reinforcement learning. Directly ranking same-query rollouts provides a more suitable relative reward signal, but existing ranking-based reward methods can incur substantial judging cost. We introduce RankBuffer, which maintains an ordered, query-specific buffer of previously judged responses as a reusable quality scale. Each rollout is first inserted into an anchor interval through an independent coarse judgment, after which only rollouts assigned to the same interval undergo local fine ranking. The resulting complete order is converted into bounded rank rewards, while boundary expansion, local refinement, and inactive-anchor pruning adapt the buffer as the policy evolves. Across four open-ended benchmarks, RankBuffer consistently outperforms all pointwise baselines. It also achieves nearly on-par performance with the strongest ranking-based reward baseline while substantially reducing judging cost. Ablations demonstrate the importance of both local fine ranking and anchor response content, while buffer analyses show that rollout-derived anchors progressively extend and refine the covered quality scale. These results establish response reuse as an effective approach to efficient relative reward construction.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yang, Z., Chen, Y., Liu, Q., Yang, W., Zhang, E., Chen, L., Wang, Q., Gao, Y., & Mao, J. (2026). RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation. https://omanscience.com/en/articles/rankbuffer-efficient-ranking-based-rewards-for-open-ended-generation
MLA 9
Yang, Zixuan, et al. "RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation." https://omanscience.com/en/articles/rankbuffer-efficient-ranking-based-rewards-for-open-ended-generation.
Chicago (author–date)
Yang, Zixuan, Yiqun Chen, Qi Liu, Wei Yang, Erhan Zhang, Liyi Chen, Qimeng Wang, Yan Gao, and Jiaxin Mao. 2026. "RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation." https://omanscience.com/en/articles/rankbuffer-efficient-ranking-based-rewards-for-open-ended-generation.
Harvard
Yang, Z., Chen, Y., Liu, Q., Yang, W., Zhang, E., Chen, L., Wang, Q., Gao, Y. and Mao, J. (2026) 'RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation', Available at: https://omanscience.com/en/articles/rankbuffer-efficient-ranking-based-rewards-for-open-ended-generation.
Vancouver
Yang Z, Chen Y, Liu Q, Yang W, Zhang E, Chen L, et al. RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation. https://omanscience.com/en/articles/rankbuffer-efficient-ranking-based-rewards-for-open-ended-generation
IEEE
Z. Yang, Y. Chen, Q. Liu, W. Yang, E. Zhang, L. Chen, Q. Wang, Y. Gao, and J. Mao, "RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation," https://omanscience.com/en/articles/rankbuffer-efficient-ranking-based-rewards-for-open-ended-generation.