Abstract
Large language models increasingly justify legal decisions by naming the statute or precedent behind a verdict, treated as evidence that the decision follows from it. We test this directly: holding case facts fixed, we substitute the named legal authority for an unrelated one and decode a model's evolving verdict from its hidden states. Across seven open-weight models (8B-70B) and four benchmarks spanning judicial and contractual reasoning, when explicitly required to justify a verdict by naming the governing authority, models name the correct one in 66.7%-100% of generations, while the verdict changing when the authority changes is far less consistent: 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Neither scale nor a purpose-built legal-reasoning model (a best-effort LoRA reproduction; Section 6) closes this gap. A red-teaming evaluation on five core models finds compliance with an adversarial instruction hidden in the case facts (73.3%-96.4%) exceeds verdict-swap sensitivity by a wide margin, holding without exception across model rankings. Naming a legal authority is thus a poor proxy for a verdict's dependence on it, while the same verdict remains separately vulnerable to adversarial manipulation. Both findings replicate across checks ruling out prompt-wording noise and confounded sampling, and bear directly on the use of generated legal explanations as compliance or audit artefacts.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Sadhu, S., Arora, S., & Seth, P. (2026). Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness. https://omanscience.com/en/articles/cited-but-not-consulted-a-counterfactual-audit-of-legal-chain-of-thought-faithfulness
MLA 9
Sadhu, Saisab, et al. "Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness." https://omanscience.com/en/articles/cited-but-not-consulted-a-counterfactual-audit-of-legal-chain-of-thought-faithfulness.
Chicago (author–date)
Sadhu, Saisab, Shreeyans Arora, and Pratinav Seth. 2026. "Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness." https://omanscience.com/en/articles/cited-but-not-consulted-a-counterfactual-audit-of-legal-chain-of-thought-faithfulness.
Harvard
Sadhu, S., Arora, S. and Seth, P. (2026) 'Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness', Available at: https://omanscience.com/en/articles/cited-but-not-consulted-a-counterfactual-audit-of-legal-chain-of-thought-faithfulness.
Vancouver
Sadhu S, Arora S, Seth P. Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness. https://omanscience.com/en/articles/cited-but-not-consulted-a-counterfactual-audit-of-legal-chain-of-thought-faithfulness
IEEE
S. Sadhu, S. Arora, and P. Seth, "Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness," https://omanscience.com/en/articles/cited-but-not-consulted-a-counterfactual-audit-of-legal-chain-of-thought-faithfulness.