Abstract
Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Alexander, D. R. K., & Kling, C. L. (2026). Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models. https://omanscience.com/en/articles/validity-without-ground-truth-what-stated-preference-economics-offers-the-evaluation-of-language-models
MLA 9
Alexander, Daniel Robert Kling, and Catherine Louise Kling. "Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models." https://omanscience.com/en/articles/validity-without-ground-truth-what-stated-preference-economics-offers-the-evaluation-of-language-models.
Chicago (author–date)
Alexander, Daniel Robert Kling, and Catherine Louise Kling. 2026. "Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models." https://omanscience.com/en/articles/validity-without-ground-truth-what-stated-preference-economics-offers-the-evaluation-of-language-models.
Harvard
Alexander, D. R. K. and Kling, C. L. (2026) 'Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models', Available at: https://omanscience.com/en/articles/validity-without-ground-truth-what-stated-preference-economics-offers-the-evaluation-of-language-models.
Vancouver
Alexander DRK, Kling CL. Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models. https://omanscience.com/en/articles/validity-without-ground-truth-what-stated-preference-economics-offers-the-evaluation-of-language-models
IEEE
D. R. K. Alexander, and C. L. Kling, "Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models," https://omanscience.com/en/articles/validity-without-ground-truth-what-stated-preference-economics-offers-the-evaluation-of-language-models.