Abstract
Chain-of-Thought (CoT) faithfulness detectors are widely used to audit reasoning models, yet a detector is itself a predictor whose verdicts are treated as stable properties. We ask whether a detector is faithful to itself under distribution shift. We formalize meta-faithfulness as an invariance principle: a valid detector must return identical verdicts on traces that differ only by transformations preserving ground-truth faithfulness. We prove three results: (i) no detector using only intervention-response profiles can separate faithful from epiphenomenal mechanisms with identical signatures; (ii) any detector relying on shift-sensitive features violates invariance at a rate independent of its in-distribution accuracy; (iii) an asymptotic certified selective-risk guarantee enables confident abstention. We operationalize the principle in FaithShift, a stress-test protocol spanning ten shift axes, and propose SIFT, a hidden-state trajectory detector trained with cross-environment invariance objectives and certified abstention. Across 14,996 traces, four domains, and eight models, three findings emerge. First, transfer collapse is real: all existing detectors show gaps $\geq 0.15$ AUROC. Second, the dominant bottleneck is sampling stochasticity, not shift: over 80% of detector instability stems from random seed variation, falsifying our preregistered prediction that shift-attributable violations exceed 0.25. Third, SIFT cuts invariance violations by 64% over the best single-seed baseline, but a four-seed ensemble of any detector narrows the margin to 0.01 (indistinguishable at matched coverage, $p=0.21$), and SIFT needs a 51% abstention rate. Cross-model transfer degrades from within-family to cross-family to open-weight-to-API, partly closed by multi-model training. We offer a framework for auditing auditors: the real barrier is detector variance, not distribution shift.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Mohammad, N. I. S., Shammo, M. B. A. Z., Siddiki, H., Hasan, M., Sheikh, M. F., & Habib, J. (2026). SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift. https://omanscience.com/en/articles/sift-robust-meta-faithfulness-verification-of-chain-of-thought-reasoning-under-distribution-shift
MLA 9
Mohammad, Noor Islam S., et al. "SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift." https://omanscience.com/en/articles/sift-robust-meta-faithfulness-verification-of-chain-of-thought-reasoning-under-distribution-shift.
Chicago (author–date)
Mohammad, Noor Islam S., Md. Basim Al Zabir Shammo, Hasan Siddiki, Mahmudul Hasan, Md. Faisal Sheikh, and Jakaria Habib. 2026. "SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift." https://omanscience.com/en/articles/sift-robust-meta-faithfulness-verification-of-chain-of-thought-reasoning-under-distribution-shift.
Harvard
Mohammad, N. I. S., Shammo, M. B. A. Z., Siddiki, H., Hasan, M., Sheikh, M. F. and Habib, J. (2026) 'SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift', Available at: https://omanscience.com/en/articles/sift-robust-meta-faithfulness-verification-of-chain-of-thought-reasoning-under-distribution-shift.
Vancouver
Mohammad NIS, Shammo MBAZ, Siddiki H, Hasan M, Sheikh MF, Habib J. SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift. https://omanscience.com/en/articles/sift-robust-meta-faithfulness-verification-of-chain-of-thought-reasoning-under-distribution-shift
IEEE
N. I. S. Mohammad, M. B. A. Z. Shammo, H. Siddiki, M. Hasan, M. F. Sheikh, and J. Habib, "SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift," https://omanscience.com/en/articles/sift-robust-meta-faithfulness-verification-of-chain-of-thought-reasoning-under-distribution-shift.