Abstract
Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is done by the perplexity. Diffusion and flow-based language models can only provide a likelihood bound, whose tightness differs between model families. Sample-based substitutes such as generative perplexity with entropy do not consider the distribution fit. We propose SOL, a distance between text distributions. Each sequence is represented by the empirical measure of its hidden states under a fixed transformer and the distributions of these measures are compared by the double sliced Wasserstein distance. We prove that SOL is a metric if the transformer is injective. Experiments show that SOL detects distributional failures, recovers expected model trends, and provides stable sample-based estimates. We put forward SOL to fill the gap in the current evaluation protocol used for non auto-regressive models. As a first step we use SOL to re-evaluate a variety of models trained on OpenWebText.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Kornhardt, G., Piening, M., Chemseddine, J., & Steidl, G. (2026). SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics. https://omanscience.com/en/articles/sol-measuring-gaps-between-text-distributions-by-double-sliced-wasserstein-metrics
MLA 9
Kornhardt, Gregor, et al. "SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics." https://omanscience.com/en/articles/sol-measuring-gaps-between-text-distributions-by-double-sliced-wasserstein-metrics.
Chicago (author–date)
Kornhardt, Gregor, Moritz Piening, Jannis Chemseddine, and Gabriele Steidl. 2026. "SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics." https://omanscience.com/en/articles/sol-measuring-gaps-between-text-distributions-by-double-sliced-wasserstein-metrics.
Harvard
Kornhardt, G., Piening, M., Chemseddine, J. and Steidl, G. (2026) 'SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics', Available at: https://omanscience.com/en/articles/sol-measuring-gaps-between-text-distributions-by-double-sliced-wasserstein-metrics.
Vancouver
Kornhardt G, Piening M, Chemseddine J, Steidl G. SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics. https://omanscience.com/en/articles/sol-measuring-gaps-between-text-distributions-by-double-sliced-wasserstein-metrics
IEEE
G. Kornhardt, M. Piening, J. Chemseddine, and G. Steidl, "SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics," https://omanscience.com/en/articles/sol-measuring-gaps-between-text-distributions-by-double-sliced-wasserstein-metrics.