Abstract
Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Li, Z., Tian, W., Xu, X., Xiao, Y., Ban, Y., Peng, Y., & Zhuang, F. (2026). Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation. https://omanscience.com/en/articles/dual-channel-robust-group-relative-policy-optimization-via-advantage-and-sequence-weight-estimation
MLA 9
Li, Zhongyi, et al. "Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation." https://omanscience.com/en/articles/dual-channel-robust-group-relative-policy-optimization-via-advantage-and-sequence-weight-estimation.
Chicago (author–date)
Li, Zhongyi, Wan Tian, Xiang Xu, Yutian Xiao, Yikun Ban, Yijie Peng, and Fuzhen Zhuang. 2026. "Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation." https://omanscience.com/en/articles/dual-channel-robust-group-relative-policy-optimization-via-advantage-and-sequence-weight-estimation.
Harvard
Li, Z., Tian, W., Xu, X., Xiao, Y., Ban, Y., Peng, Y. and Zhuang, F. (2026) 'Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation', Available at: https://omanscience.com/en/articles/dual-channel-robust-group-relative-policy-optimization-via-advantage-and-sequence-weight-estimation.
Vancouver
Li Z, Tian W, Xu X, Xiao Y, Ban Y, Peng Y, et al. Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation. https://omanscience.com/en/articles/dual-channel-robust-group-relative-policy-optimization-via-advantage-and-sequence-weight-estimation
IEEE
Z. Li, W. Tian, X. Xu, Y. Xiao, Y. Ban, Y. Peng, and F. Zhuang, "Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation," https://omanscience.com/en/articles/dual-channel-robust-group-relative-policy-optimization-via-advantage-and-sequence-weight-estimation.