الملخص
Rubric-based reinforcement learning (RL) provides interpretable rewards for aligning large language models (LLMs) by evaluating responses against query-specific evaluation criteria. To construct rubrics at scale, a straightforward approach to LLM-based rubric generation is to prompt an LLM to generate a rubric directly from the query. However, rubrics directly generated by LLMs are vulnerable to reward hacking, since omitted or underspecified criteria allow the policy to obtain high rubric rewards with low-quality responses. Existing LLM-based rubric generation methods improve the granularity and coverage of the generated criteria but do not proactively guard against reward hacking. To address this limitation, we propose RubricArmor, an adversarial framework that exposes and mitigates potential reward hacking at the rubric generation stage before it occurs in subsequent RL. Specifically, RubricArmor performs adversarial evolution, in which an attack step and a repair step alternate over multiple rounds. The attack step simulates the reward hacking of the policy by constructing adversarial responses that satisfy the current rubric but fail to properly complete the task. The repair step then revises the rubric to detect the response defects exposed by the attack step while preserving other valid criteria. Extensive experiments demonstrate that RubricArmor outperforms competitive rubric generation baselines and translates into more effective downstream rubric-based RL.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Yang, H., Zhang, Y., Pan, L., Fan, J., Wang, M., Zhang, K., Shao, S., Wang, S., Lu, Y., Zheng, C., & Wang, H. (2026). RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation. https://omanscience.com/ar/articles/rubricarmor-adversarial-evolution-improves-llm-based-rubric-generation
MLA 9
Yang, Haocheng, et al. "RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation." https://omanscience.com/ar/articles/rubricarmor-adversarial-evolution-improves-llm-based-rubric-generation.
شيكاغو (المؤلف–التاريخ)
Yang, Haocheng, Yuchao Zhang, Licheng Pan, Jiajun Fan, Maolin Wang, Kangning Zhang, Shuai Shao, Shijian Wang, Yuan Lu, Chunyuan Zheng, and Hao Wang. 2026. "RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation." https://omanscience.com/ar/articles/rubricarmor-adversarial-evolution-improves-llm-based-rubric-generation.
هارفارد
Yang, H., Zhang, Y., Pan, L., Fan, J., Wang, M., Zhang, K., Shao, S., Wang, S., Lu, Y., Zheng, C. and Wang, H. (2026) 'RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation', Available at: https://omanscience.com/ar/articles/rubricarmor-adversarial-evolution-improves-llm-based-rubric-generation.
فانكوفر
Yang H, Zhang Y, Pan L, Fan J, Wang M, Zhang K, et al. RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation. https://omanscience.com/ar/articles/rubricarmor-adversarial-evolution-improves-llm-based-rubric-generation
IEEE
H. Yang, Y. Zhang, L. Pan, J. Fan, M. Wang, K. Zhang, S. Shao, S. Wang, Y. Lu, C. Zheng, and H. Wang, "RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation," https://omanscience.com/ar/articles/rubricarmor-adversarial-evolution-improves-llm-based-rubric-generation.