Preprint Open access
Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling
Multiple-choice benchmarks are cheap to grade and are running out of room, and the standard remedy, writing harder items, is slow and repeated for every benchmark. A saturated benchmark still holds a harder task. Each question's wrong options are written for that question alone, so a model can score by eliminating a fe …