الملخص
Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.
الكلمات المفتاحية
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Kim, Y., & Kwak, N. (2026). Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization. https://omanscience.com/ar/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization
MLA 9
Kim, Yun, and Nojun Kwak. "Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization." https://omanscience.com/ar/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.
شيكاغو (المؤلف–التاريخ)
Kim, Yun, and Nojun Kwak. 2026. "Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization." https://omanscience.com/ar/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.
هارفارد
Kim, Y. and Kwak, N. (2026) 'Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization', Available at: https://omanscience.com/ar/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.
فانكوفر
Kim Y, Kwak N. Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization. https://omanscience.com/ar/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization
IEEE
Y. Kim, and N. Kwak, "Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization," https://omanscience.com/ar/articles/advancing-entropy-level-credit-assignment-in-rlvr-via-proximal-entropy-policy-optimization.