Abstract
LLM-as-a-Judge is increasingly used to evaluate policy responses on open-ended tasks that lack ground-truth answers. Existing work often directly converts the resulting judgments into reward signals for policy training, paying limited attention to intrinsic judgment quality and largely restricting the use of Judges to training-time supervision. We systematically investigate judgment quality and downstream utility by examining both how judgments are elicited and how they are used. For judgment elicitation, we vary the Judge protocol along three dimensions: verdict granularity, critique usage, and evaluation batching. For judgment usage, beyond policy training, we extend Judge to test-time inference through Best-of-N selection, Judge-guided revision, and beam search. We find that, (i) Surprisingly, judgment quality and downstream utility do not always align. (ii) Judge protocol design substantially affects both intrinsic judgment quality and downstream utility. (iii) Judge guidance effectively converts test-time compute into performance gains, with benefits varying across inference strategies. Our results call for a multifaceted evaluation of LLM Judges on open-ended tasks, encompassing intrinsic judgment quality, and downstream utility.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhang, Z., Li, L., Tan, X., Zeng, Y., Shan, Z., Li, Y., & Ren, K. (2026). From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks. https://omanscience.com/en/articles/from-judgment-quality-to-downstream-utility-rethinking-llm-as-a-judge-for-open-ended-tasks
MLA 9
Zhang, Zheng, et al. "From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks." https://omanscience.com/en/articles/from-judgment-quality-to-downstream-utility-rethinking-llm-as-a-judge-for-open-ended-tasks.
Chicago (author–date)
Zhang, Zheng, Lufei Li, Xinyue Tan, Yuanhao Zeng, Ziwei Shan, Yexin Li, and Kan Ren. 2026. "From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks." https://omanscience.com/en/articles/from-judgment-quality-to-downstream-utility-rethinking-llm-as-a-judge-for-open-ended-tasks.
Harvard
Zhang, Z., Li, L., Tan, X., Zeng, Y., Shan, Z., Li, Y. and Ren, K. (2026) 'From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks', Available at: https://omanscience.com/en/articles/from-judgment-quality-to-downstream-utility-rethinking-llm-as-a-judge-for-open-ended-tasks.
Vancouver
Zhang Z, Li L, Tan X, Zeng Y, Shan Z, Li Y, et al. From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks. https://omanscience.com/en/articles/from-judgment-quality-to-downstream-utility-rethinking-llm-as-a-judge-for-open-ended-tasks
IEEE
Z. Zhang, L. Li, X. Tan, Y. Zeng, Z. Shan, Y. Li, and K. Ren, "From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks," https://omanscience.com/en/articles/from-judgment-quality-to-downstream-utility-rethinking-llm-as-a-judge-for-open-ended-tasks.