الملخص
While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbf{adaptation imbalance}: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that \textbf{learning where to adapt does not ensure that adaptation gains are well balanced}. This motivates \textbf{LoRA-Norm}, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Tian, Z., Chen, Y., Han, Z., Wang, H., & Liao, L. (2026). Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA. https://omanscience.com/ar/articles/learn-the-directions-normalize-the-gains-post-training-normalization-for-lora
MLA 9
Tian, Zailong, et al. "Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA." https://omanscience.com/ar/articles/learn-the-directions-normalize-the-gains-post-training-normalization-for-lora.
شيكاغو (المؤلف–التاريخ)
Tian, Zailong, Yanzhe Chen, Zhuoheng Han, Houfeng Wang, and Lizi Liao. 2026. "Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA." https://omanscience.com/ar/articles/learn-the-directions-normalize-the-gains-post-training-normalization-for-lora.
هارفارد
Tian, Z., Chen, Y., Han, Z., Wang, H. and Liao, L. (2026) 'Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA', Available at: https://omanscience.com/ar/articles/learn-the-directions-normalize-the-gains-post-training-normalization-for-lora.
فانكوفر
Tian Z, Chen Y, Han Z, Wang H, Liao L. Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA. https://omanscience.com/ar/articles/learn-the-directions-normalize-the-gains-post-training-normalization-for-lora
IEEE
Z. Tian, Y. Chen, Z. Han, H. Wang, and L. Liao, "Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA," https://omanscience.com/ar/articles/learn-the-directions-normalize-the-gains-post-training-normalization-for-lora.