Preprint Open access
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable tran …