Abstract
The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation in a high-resource translation direction for building high-quality NLP resources in low-resource language/domain pairs.
Keywords
Subject
Publication details
- DOI
- 10.18653/v1/2025.findings-emnlp.1053
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Knafou, J., Mottin, L., Mottaz, A., Flament, A., & Ruch, P. (2026). TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling. https://doi.org/10.18653/v1/2025.findings-emnlp.1053
MLA 9
Knafou, Julien, et al. "TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling." https://doi.org/10.18653/v1/2025.findings-emnlp.1053.
Chicago (author–date)
Knafou, Julien, Luc Mottin, Anaïs Mottaz, Alexandre Flament, and Patrick Ruch. 2026. "TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling." https://doi.org/10.18653/v1/2025.findings-emnlp.1053.
Harvard
Knafou, J., Mottin, L., Mottaz, A., Flament, A. and Ruch, P. (2026) 'TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling', doi:10.18653/v1/2025.findings-emnlp.1053.
Vancouver
Knafou J, Mottin L, Mottaz A, Flament A, Ruch P. TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling. doi:10.18653/v1/2025.findings-emnlp.1053
IEEE
J. Knafou, L. Mottin, A. Mottaz, A. Flament, and P. Ruch, "TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling," doi: 10.18653/v1/2025.findings-emnlp.1053.