الملخص
Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at https://github.com/Estrellajer/CoDe-LoRA.
الكلمات المفتاحية
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Liu, M., Fang, Q., & He, Y. (2026). CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling. https://omanscience.com/ar/articles/code-lora-mitigating-the-orthogonality-dilemma-in-continual-learning-of-llms-via-knowledge-consolidation-and-decoupling
MLA 9
Liu, Maoqi, et al. "CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling." https://omanscience.com/ar/articles/code-lora-mitigating-the-orthogonality-dilemma-in-continual-learning-of-llms-via-knowledge-consolidation-and-decoupling.
شيكاغو (المؤلف–التاريخ)
Liu, Maoqi, Quan Fang, and Yufei He. 2026. "CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling." https://omanscience.com/ar/articles/code-lora-mitigating-the-orthogonality-dilemma-in-continual-learning-of-llms-via-knowledge-consolidation-and-decoupling.
هارفارد
Liu, M., Fang, Q. and He, Y. (2026) 'CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling', Available at: https://omanscience.com/ar/articles/code-lora-mitigating-the-orthogonality-dilemma-in-continual-learning-of-llms-via-knowledge-consolidation-and-decoupling.
فانكوفر
Liu M, Fang Q, He Y. CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling. https://omanscience.com/ar/articles/code-lora-mitigating-the-orthogonality-dilemma-in-continual-learning-of-llms-via-knowledge-consolidation-and-decoupling
IEEE
M. Liu, Q. Fang, and Y. He, "CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling," https://omanscience.com/ar/articles/code-lora-mitigating-the-orthogonality-dilemma-in-continual-learning-of-llms-via-knowledge-consolidation-and-decoupling.