الملخص
تمت ترجمة أجزاء من هذه الصفحة آلياً وقد تحتوي على أخطاء.
Continual fine-tuning is essential for large language models (LLMs) to dynamically adapt to real-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs' general-purpose knowledge. Although existing methods, such as orthogonal gradient projection, mitigate the forgetting across various fine-tuning tasks, they fundamentally fail to preserve pre-training LLMs' inherent general-purpose knowledge because the original data and gradients of off-the-shelf pre-training LLMs required by these methods are strictly unknown and highly diverse. To bridge this critical gap, we propose EoupCT, a novel framework designed to Estimate and Orthogonalize Unknown Pre-training gradients for Continual LLM fine-Tuning. Specifically, EoupCT estimates pre-training gradients by dynamically generating pseudo data that is most susceptible to forgetting for new tasks through a learnable soft prompt equipped with Gumbel-Softmax relaxation. Furthermore, we formulate a multi-objective optimization problem and introduce a first-order efficient Pareto optimizer that jointly optimizes LLM parameters and the soft prompt, rigorously enforcing orthogonality between new task updates and the estimated pre-training gradients. Extensive experiments across multiple LLMs demonstrate that EoupCT effectively preserves both task-specific proficiency and inherent general-purpose knowledge, successfully mitigating the catastrophic forgetting.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Wang, B., Li, C., Cai, X. Q., Wu, L. Y., Li, X., Niu, G., & Sugiyama, M. (2026). تقدير تدرجات ما قبل التدريب المجهولة وتعامدها للضبط الدقيق المستمر للنماذج اللغوية الكبيرة. https://omanscience.com/ar/articles/estimating-and-orthogonalizing-unknown-pre-training-gradients-for-continual-fine-tuning-of-large-language-models
MLA 9
Wang, Bing, et al. "تقدير تدرجات ما قبل التدريب المجهولة وتعامدها للضبط الدقيق المستمر للنماذج اللغوية الكبيرة." https://omanscience.com/ar/articles/estimating-and-orthogonalizing-unknown-pre-training-gradients-for-continual-fine-tuning-of-large-language-models.
شيكاغو (المؤلف–التاريخ)
Wang, Bing, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, and Masashi Sugiyama. 2026. "تقدير تدرجات ما قبل التدريب المجهولة وتعامدها للضبط الدقيق المستمر للنماذج اللغوية الكبيرة." https://omanscience.com/ar/articles/estimating-and-orthogonalizing-unknown-pre-training-gradients-for-continual-fine-tuning-of-large-language-models.
هارفارد
Wang, B., Li, C., Cai, X. Q., Wu, L. Y., Li, X., Niu, G. and Sugiyama, M. (2026) 'تقدير تدرجات ما قبل التدريب المجهولة وتعامدها للضبط الدقيق المستمر للنماذج اللغوية الكبيرة', Available at: https://omanscience.com/ar/articles/estimating-and-orthogonalizing-unknown-pre-training-gradients-for-continual-fine-tuning-of-large-language-models.
فانكوفر
Wang B, Li C, Cai XQ, Wu LY, Li X, Niu G, et al. تقدير تدرجات ما قبل التدريب المجهولة وتعامدها للضبط الدقيق المستمر للنماذج اللغوية الكبيرة. https://omanscience.com/ar/articles/estimating-and-orthogonalizing-unknown-pre-training-gradients-for-continual-fine-tuning-of-large-language-models
IEEE
B. Wang, C. Li, X. Q. Cai, L. Y. Wu, X. Li, G. Niu, and M. Sugiyama, "تقدير تدرجات ما قبل التدريب المجهولة وتعامدها للضبط الدقيق المستمر للنماذج اللغوية الكبيرة," https://omanscience.com/ar/articles/estimating-and-orthogonalizing-unknown-pre-training-gradients-for-continual-fine-tuning-of-large-language-models.