الملخص
Self-supervised pretraining learns representations that are broadly transferable across downstream tasks, yet direct fine-tuning can be suboptimal due to misalignment between self-supervised and downstream task objectives, potentially degrading pretrained features beneficial to the downstream task. The BiSSL framework addressed this by introducing a transitional training stage formulated as a bilevel optimization problem, in which the downstream task objective guides the self-supervised learning process in refining pretrained representations to better facilitate subsequent fine-tuning. However, BiSSL relies on conventional bilevel optimization solving techniques whose costly implicit hypergradient approximations render the method increasingly impractical for contemporary model architectures. To make it efficient and scalable, we introduce BiSSLight, which combines M-FAC-based implicit gradient approximation with parameter-efficient fine-tuning via LoRA, enabling efficient application at larger scales that were previously impractical. Evaluation across multiple downstream tasks and contemporary model architectures shows that BiSSLight consistently improves downstream performance, with gains becoming more pronounced as model size increases despite stronger baselines. The method is highly computationally efficient, reducing computation time by more than a factor of ten compared to its predecessor on a ViT-H backbone.
الكلمات المفتاحية
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Zakarias, G. W., & Tan, Z. H. (2026). A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning. https://omanscience.com/ar/articles/a-light-bilevel-refinement-aligns-self-supervised-representations-for-stronger-task-specific-learning
MLA 9
Zakarias, Gustav Wagner, and Zheng-Hua Tan. "A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning." https://omanscience.com/ar/articles/a-light-bilevel-refinement-aligns-self-supervised-representations-for-stronger-task-specific-learning.
شيكاغو (المؤلف–التاريخ)
Zakarias, Gustav Wagner, and Zheng-Hua Tan. 2026. "A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning." https://omanscience.com/ar/articles/a-light-bilevel-refinement-aligns-self-supervised-representations-for-stronger-task-specific-learning.
هارفارد
Zakarias, G. W. and Tan, Z. H. (2026) 'A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning', Available at: https://omanscience.com/ar/articles/a-light-bilevel-refinement-aligns-self-supervised-representations-for-stronger-task-specific-learning.
فانكوفر
Zakarias GW, Tan ZH. A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning. https://omanscience.com/ar/articles/a-light-bilevel-refinement-aligns-self-supervised-representations-for-stronger-task-specific-learning
IEEE
G. W. Zakarias, and Z. H. Tan, "A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning," https://omanscience.com/ar/articles/a-light-bilevel-refinement-aligns-self-supervised-representations-for-stronger-task-specific-learning.