الملخص
The reasoning trajectory of a Large Language Model (LLM) is often treated as a verbalized description of its internal reasoning. However, such trajectories can be unfaithful: a model may rely on shortcuts to reach an answer and then post-rationalize the decision with a seemingly coherent chain of thought. Detecting this shortcut reasoning is challenging because existing monitors and verifiers mainly inspect textual traces or final outcomes, rather than how the model's belief in its answer develops during generation. We introduce ConfLens, a framework that tracks the evolution of confidence in the final answer throughout reasoning. Across three shortcut reasoning settings, we observe a common pattern of premature confidence, where shortcut samples become highly confident in the final answer at early reasoning stages. Existing confidence estimation methods, however, show limited generalizability, reliability, or efficiency for detecting this behavior. We therefore propose the Distributional Answer Commitment Score (DACS), a distributional confidence estimator that measures the entropy of the model's probability distribution over answer commitment at each reasoning step. DACS captures how concentrated the model's answer belief is without requiring ground-truth answers or task-specific verifiers. We further convert ConfLens detection results into interpretable signals for reward models to reduce their preference for shortcut reasoning. Experiments on mathematical and code reasoning tasks show that ConfLens with DACS improves shortcut reasoning detection by over 4.3% F1 compared with strong baselines and reduces the mismatch between faithfulness and correctness in reward model preferences.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Zhang, Z., Liu, J., Li, C., Shen, C., Liu, X., Shen, C., Ye, J., Liu, Z., & Patras, I. (2026). When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment. https://omanscience.com/ar/articles/when-confidence-rises-too-early-detecting-shortcut-reasoning-via-premature-answer-commitment
MLA 9
Zhang, Zhaohan, et al. "When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment." https://omanscience.com/ar/articles/when-confidence-rises-too-early-detecting-shortcut-reasoning-via-premature-answer-commitment.
شيكاغو (المؤلف–التاريخ)
Zhang, Zhaohan, Junjie Liu, Chengzhengxu Li, Chen Shen, Xiaoming Liu, Chao Shen, Jieping Ye, Ziquan Liu, and Ioannis Patras. 2026. "When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment." https://omanscience.com/ar/articles/when-confidence-rises-too-early-detecting-shortcut-reasoning-via-premature-answer-commitment.
هارفارد
Zhang, Z., Liu, J., Li, C., Shen, C., Liu, X., Shen, C., Ye, J., Liu, Z. and Patras, I. (2026) 'When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment', Available at: https://omanscience.com/ar/articles/when-confidence-rises-too-early-detecting-shortcut-reasoning-via-premature-answer-commitment.
فانكوفر
Zhang Z, Liu J, Li C, Shen C, Liu X, Shen C, et al. When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment. https://omanscience.com/ar/articles/when-confidence-rises-too-early-detecting-shortcut-reasoning-via-premature-answer-commitment
IEEE
Z. Zhang, J. Liu, C. Li, C. Shen, X. Liu, C. Shen, J. Ye, Z. Liu, and I. Patras, "When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment," https://omanscience.com/ar/articles/when-confidence-rises-too-early-detecting-shortcut-reasoning-via-premature-answer-commitment.