الملخص
تمت ترجمة أجزاء من هذه الصفحة آلياً وقد تحتوي على أخطاء.
Fine-tuning pre-trained models on specialized tasks with scarce data is central to modern deep learning. Despite its empirical success, theoretical understanding of fine-tuning remains limited. We introduce a Gaussian multi-index setting to study fine-tuning from pre-trained weights, where the teacher network has $m+1$ features, $m$ of which are learned during pre-training and one of which must be learned during fine-tuning. For two-layer ReLU networks, we show that two-timescale training, i.e., updating the outer weights infinitely faster than the hidden ones, learns the new task-specific feature while preserving the pre-trained ones in the model representation. Moreover, only $\mathcal{O}(d)$ fine-tuning samples are required for this recovery, independently of the number of pre-trained features. In contrast, with random initialization, the same number of samples is insufficient to recover the target parameters. Our results therefore demonstrate that pre-training can induce an implicit bias with a clear statistical advantage over random initialization, enabling feature learning from scarce fine-tuning data.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Boursier, E., & Flammarion, N. (2026). الضبط الدقيق ثنائي المقياس الزمني يتعلم ميزات جديدة بإثبات لشبكات ReLU ذات الطبقتين. https://omanscience.com/ar/articles/two-timescale-fine-tuning-provably-learns-new-features-for-two-layer-relu-networks
MLA 9
Boursier, Etienne, and Nicolas Flammarion. "الضبط الدقيق ثنائي المقياس الزمني يتعلم ميزات جديدة بإثبات لشبكات ReLU ذات الطبقتين." https://omanscience.com/ar/articles/two-timescale-fine-tuning-provably-learns-new-features-for-two-layer-relu-networks.
شيكاغو (المؤلف–التاريخ)
Boursier, Etienne, and Nicolas Flammarion. 2026. "الضبط الدقيق ثنائي المقياس الزمني يتعلم ميزات جديدة بإثبات لشبكات ReLU ذات الطبقتين." https://omanscience.com/ar/articles/two-timescale-fine-tuning-provably-learns-new-features-for-two-layer-relu-networks.
هارفارد
Boursier, E. and Flammarion, N. (2026) 'الضبط الدقيق ثنائي المقياس الزمني يتعلم ميزات جديدة بإثبات لشبكات ReLU ذات الطبقتين', Available at: https://omanscience.com/ar/articles/two-timescale-fine-tuning-provably-learns-new-features-for-two-layer-relu-networks.
فانكوفر
Boursier E, Flammarion N. الضبط الدقيق ثنائي المقياس الزمني يتعلم ميزات جديدة بإثبات لشبكات ReLU ذات الطبقتين. https://omanscience.com/ar/articles/two-timescale-fine-tuning-provably-learns-new-features-for-two-layer-relu-networks
IEEE
E. Boursier, and N. Flammarion, "الضبط الدقيق ثنائي المقياس الزمني يتعلم ميزات جديدة بإثبات لشبكات ReLU ذات الطبقتين," https://omanscience.com/ar/articles/two-timescale-fine-tuning-provably-learns-new-features-for-two-layer-relu-networks.