Abstract
Generative Information Retrieval (GIR) has emerged as a transformative paradigm, shifting document retrieval from a traditional "retrieve-and-rank" workflow to sequence-to-sequence generation, where a model directly predicts document identifiers (DocIDs). While the semantic design of these DocIDs is known to be critical for performance, a fundamental question remains under-explored: what makes a good DocID? Current approaches rely heavily on computationally expensive downstream evaluations, hindering systematic analysis and rapid iteration. In this work, we address this challenge by presenting a comprehensive study on the properties, metrics, and trade-offs that define effective numerical DocIDs. Specifically, our contributions are threefold: First, we propose a unified framework that unifies Product Quantization (PQ) and Residual Quantization (RQ), and their hybrid variants within a single design space. This enables us to systematically study key DocID properties, such as hierarchy versus parallelism, as well as the impact of hyperparameters like DocID length and codebook size. Second, we define a suite of training-free, intrinsic metrics, to quantify DocID quality and evaluate structural fidelity without the overhead of full model training. Through extensive experiments on MS MARCO 300K and NQ320K, we analyze how these structural properties influence retrieval effectiveness.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Allal, A., Randrianarivo, H., & Lamprier, S. (2026). A Systematic Study of Semantic ID Spaces for Generative Information Retrieval. https://omanscience.com/en/articles/a-systematic-study-of-semantic-id-spaces-for-generative-information-retrieval
MLA 9
Allal, Alexia, et al. "A Systematic Study of Semantic ID Spaces for Generative Information Retrieval." https://omanscience.com/en/articles/a-systematic-study-of-semantic-id-spaces-for-generative-information-retrieval.
Chicago (author–date)
Allal, Alexia, Hicham Randrianarivo, and Sylvain Lamprier. 2026. "A Systematic Study of Semantic ID Spaces for Generative Information Retrieval." https://omanscience.com/en/articles/a-systematic-study-of-semantic-id-spaces-for-generative-information-retrieval.
Harvard
Allal, A., Randrianarivo, H. and Lamprier, S. (2026) 'A Systematic Study of Semantic ID Spaces for Generative Information Retrieval', Available at: https://omanscience.com/en/articles/a-systematic-study-of-semantic-id-spaces-for-generative-information-retrieval.
Vancouver
Allal A, Randrianarivo H, Lamprier S. A Systematic Study of Semantic ID Spaces for Generative Information Retrieval. https://omanscience.com/en/articles/a-systematic-study-of-semantic-id-spaces-for-generative-information-retrieval
IEEE
A. Allal, H. Randrianarivo, and S. Lamprier, "A Systematic Study of Semantic ID Spaces for Generative Information Retrieval," https://omanscience.com/en/articles/a-systematic-study-of-semantic-id-spaces-for-generative-information-retrieval.