Preprint Open access
The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)
How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational esti …