الملخص
Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is not possible in decoder-only LLMs because of varying multilingual tokenization. However, a growing amount of research suggests that even in LLMs, higher cross-lingual representational alignment leads to improved cross-lingual transfer. In this paper, we propose a novel approach to reimagine cross-lingual contrastive learning given the architectural constraints of modern LLMs. Rather than applying an auxiliary alignment loss on hidden states, we propose using the outputs of the mixture-of-experts (MoE) routers as the target for alignment. Router outputs lend themselves better to pooling over many tokens, enabling more reliable cross-lingual comparisons at the sequence-level. Controlled continual pre-training experiments on four open-source MoEs show that incorporating this routing loss also aligns the underlying hidden representations across languages. Most importantly, this loss improves multilingual performance on our diverse evaluation suite, demonstrating the potential of cross-lingual MoE router alignment.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Bandarkar, L., Peng, C., Ahmed, A. H., Khandelwal, A., & Peng, N. (2026). Cross-Lingual Alignment for Decoder-Only Models using MoE Routers. https://omanscience.com/ar/articles/cross-lingual-alignment-for-decoder-only-models-using-moe-routers
MLA 9
Bandarkar, Lucas, et al. "Cross-Lingual Alignment for Decoder-Only Models using MoE Routers." https://omanscience.com/ar/articles/cross-lingual-alignment-for-decoder-only-models-using-moe-routers.
شيكاغو (المؤلف–التاريخ)
Bandarkar, Lucas, Clark Peng, Ahmed Haj Ahmed, Aditi Khandelwal, and Nanyun Peng. 2026. "Cross-Lingual Alignment for Decoder-Only Models using MoE Routers." https://omanscience.com/ar/articles/cross-lingual-alignment-for-decoder-only-models-using-moe-routers.
هارفارد
Bandarkar, L., Peng, C., Ahmed, A. H., Khandelwal, A. and Peng, N. (2026) 'Cross-Lingual Alignment for Decoder-Only Models using MoE Routers', Available at: https://omanscience.com/ar/articles/cross-lingual-alignment-for-decoder-only-models-using-moe-routers.
فانكوفر
Bandarkar L, Peng C, Ahmed AH, Khandelwal A, Peng N. Cross-Lingual Alignment for Decoder-Only Models using MoE Routers. https://omanscience.com/ar/articles/cross-lingual-alignment-for-decoder-only-models-using-moe-routers
IEEE
L. Bandarkar, C. Peng, A. H. Ahmed, A. Khandelwal, and N. Peng, "Cross-Lingual Alignment for Decoder-Only Models using MoE Routers," https://omanscience.com/ar/articles/cross-lingual-alignment-for-decoder-only-models-using-moe-routers.