Abstract
Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them. Yet existing benchmarks score only scalar quality or overall preference, so a judge can be rewarded for picking the preferred image for the wrong visual reason. We introduce VisionQ, the first benchmark built from peer-reviewed CV comparison figures that grounds every judgment in a named visual criterion: each question states the criterion, and a judge is credited only when it selects the output the authors identify as best on that criterion. We call this task criterion-conditioned visual discrimination. VisionQ comprises (1) a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures and 3,911 hand-annotated data points linking method crops to author-stated visual claims; (2) a six-axis, 51-leaf taxonomy of the visual criteria behind qualitative judgment; (3) a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity and reports accuracy per criterion; and (4) VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs, which reduces last-option predictions by 7.0pp and improves accuracy by 2.5pp on a held-out test set. Evaluating 20 open- and closed-source VLM judges, we find that the strongest reach only 63.1% accuracy (chance 32.2%) and that reliability varies sharply across criteria. Code: https://github.com/ReML-AI/visionq. Data: https://huggingface.co/datasets/visionq-anon-2026/VisionQ-1k.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Xuan, V. D., Nguyen, D. H., Dao, M. D., Giao, V. Q., Nguyen, Q. H., Hua, B. S., O'Sullivan, B., Murphy, D., & Nguyen, H. D. (2026). VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision. https://omanscience.com/en/articles/visionq-vlm-as-a-judge-taxonomy-dataset-and-benchmark-for-qualitative-analysis-in-computer-vision
MLA 9
Xuan, Vu Dinh, et al. "VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision." https://omanscience.com/en/articles/visionq-vlm-as-a-judge-taxonomy-dataset-and-benchmark-for-qualitative-analysis-in-computer-vision.
Chicago (author–date)
Xuan, Vu Dinh, Duc Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, and Hoang D. Nguyen. 2026. "VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision." https://omanscience.com/en/articles/visionq-vlm-as-a-judge-taxonomy-dataset-and-benchmark-for-qualitative-analysis-in-computer-vision.
Harvard
Xuan, V. D., Nguyen, D. H., Dao, M. D., Giao, V. Q., Nguyen, Q. H., Hua, B. S., O'Sullivan, B., Murphy, D. and Nguyen, H. D. (2026) 'VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision', Available at: https://omanscience.com/en/articles/visionq-vlm-as-a-judge-taxonomy-dataset-and-benchmark-for-qualitative-analysis-in-computer-vision.
Vancouver
Xuan VD, Nguyen DH, Dao MD, Giao VQ, Nguyen QH, Hua BS, et al. VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision. https://omanscience.com/en/articles/visionq-vlm-as-a-judge-taxonomy-dataset-and-benchmark-for-qualitative-analysis-in-computer-vision
IEEE
V. D. Xuan, D. H. Nguyen, M. D. Dao, V. Q. Giao, Q. H. Nguyen, B. S. Hua, B. O'Sullivan, D. Murphy, and H. D. Nguyen, "VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision," https://omanscience.com/en/articles/visionq-vlm-as-a-judge-taxonomy-dataset-and-benchmark-for-qualitative-analysis-in-computer-vision.