Abstract
Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of hidden representations along the time axis, with a residual low-dimension discrete bottleneck. We analyze our approach for different quantization methods, training objectives, and datasets. For different levels of compression, we evaluate the similarity between the original and reconstructed text both at the surface-level (BLEU) and at the semantic-level (LLM-based judge). Additionally, we evaluate our models on downstream question-answering and semantic text similarity benchmarks. Our approach results in compressed representations which are on par with lossless text compression algorithms at 2.24 bits per byte on web text data, while having good reconstruction and downstream task performance.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Sabolčec, V., Katharopoulos, A., & Grangier, D. (2026). Lossy Compressive Text Autoencoders. https://omanscience.com/en/articles/lossy-compressive-text-autoencoders
MLA 9
Sabolčec, Vinko, et al. "Lossy Compressive Text Autoencoders." https://omanscience.com/en/articles/lossy-compressive-text-autoencoders.
Chicago (author–date)
Sabolčec, Vinko, Angelos Katharopoulos, and David Grangier. 2026. "Lossy Compressive Text Autoencoders." https://omanscience.com/en/articles/lossy-compressive-text-autoencoders.
Harvard
Sabolčec, V., Katharopoulos, A. and Grangier, D. (2026) 'Lossy Compressive Text Autoencoders', Available at: https://omanscience.com/en/articles/lossy-compressive-text-autoencoders.
Vancouver
Sabolčec V, Katharopoulos A, Grangier D. Lossy Compressive Text Autoencoders. https://omanscience.com/en/articles/lossy-compressive-text-autoencoders
IEEE
V. Sabolčec, A. Katharopoulos, and D. Grangier, "Lossy Compressive Text Autoencoders," https://omanscience.com/en/articles/lossy-compressive-text-autoencoders.