Abstract
Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Kim, Y., & Kwak, N. (2026). Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization. https://omanscience.com/en/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization
MLA 9
Kim, Yun, and Nojun Kwak. "Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization." https://omanscience.com/en/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.
Chicago (author–date)
Kim, Yun, and Nojun Kwak. 2026. "Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization." https://omanscience.com/en/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.
Harvard
Kim, Y. and Kwak, N. (2026) 'Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization', Available at: https://omanscience.com/en/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.
Vancouver
Kim Y, Kwak N. Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization. https://omanscience.com/en/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization
IEEE
Y. Kim, and N. Kwak, "Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization," https://omanscience.com/en/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.