Abstract
Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based filtering has proven effective in selecting high-quality subsets from web-scale corpora, especially for high-resource languages, low-resource languages face challenges due to limited availability of annotated data. This work explores extending quality filtering to over 100 languages by proposing a multilingual adaptation approach that converts an existing English quality classifier into a multilingual variant. Our approach proposes training a small multi-layer perceptron on top of Transformer encoder-only model embeddings, using multilingual text as input and scores obtained from English classifiers applied to machine-translated text as labels. Our 1B, 3B and 8B scale experiments show that our approach maintains the downstream LLM benchmark performance of existing multilingual model-based filtering baselines, without harming regional and cultural knowledge benchmarks. To further evaluate cross-lingual generalization, we compare classifier scores of high-quality synthetic data and web samples, and the correlation of classifier scores with LLM-based ones, revealing that the classifier can learn the scoring criteria of its original English variant, even for languages not included in its training data.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Sabolčec, V., Messmer, B., Turki, Y., & Jaggi, M. (2026). Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection. https://omanscience.com/en/articles/adapting-english-quality-classifiers-for-multilingual-llm-pretraining-data-selection
MLA 9
Sabolčec, Vinko, et al. "Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection." https://omanscience.com/en/articles/adapting-english-quality-classifiers-for-multilingual-llm-pretraining-data-selection.
Chicago (author–date)
Sabolčec, Vinko, Bettina Messmer, Yassine Turki, and Martin Jaggi. 2026. "Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection." https://omanscience.com/en/articles/adapting-english-quality-classifiers-for-multilingual-llm-pretraining-data-selection.
Harvard
Sabolčec, V., Messmer, B., Turki, Y. and Jaggi, M. (2026) 'Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection', Available at: https://omanscience.com/en/articles/adapting-english-quality-classifiers-for-multilingual-llm-pretraining-data-selection.
Vancouver
Sabolčec V, Messmer B, Turki Y, Jaggi M. Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection. https://omanscience.com/en/articles/adapting-english-quality-classifiers-for-multilingual-llm-pretraining-data-selection
IEEE
V. Sabolčec, B. Messmer, Y. Turki, and M. Jaggi, "Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection," https://omanscience.com/en/articles/adapting-english-quality-classifiers-for-multilingual-llm-pretraining-data-selection.