الملخص
Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental "aggregation dilemma" between the accurate Sum-of-Products (SoP) and the communication-efficient Product-of-Sums (PoS) implementations. To tackle these challenges, we propose FedFit. First, to significantly reduce communication overhead, we introduce a disjoint shared vector-bank parameterization that reconstructs high-dimensional adapter matrices from two compact and disjoint global vector banks. Second, to address the aggregation dilemma, we devise an alternating optimization schedule. By cycling between decoupled single-bank updates (which allow for accurate aggregation) and joint updates corrected by a Residual Spectral Aggregation mechanism, we resolve the conflict between SoP and PoS. Additionally, we integrate blockwise quantization with client-side error feedback to further compress the transmitted vectors. Furthermore, we establish theoretical convergence guarantees for the proposed algorithm. Extensive experiments on Qwen2.5 models demonstrate that FedFit achieves perplexity performance comparable to standard federated LoRA methods, while providing compression ratios up to 100x higher.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Zou, H., Zhang, C., Yang, Y., Tian, Y., Lasaulce, S., & Debbah, M. (2026). FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization. https://omanscience.com/ar/articles/fedfit-federated-fine-tuning-of-llms-via-vector-bank-parameterization-and-quantization
MLA 9
Zou, Hang, et al. "FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization." https://omanscience.com/ar/articles/fedfit-federated-fine-tuning-of-llms-via-vector-bank-parameterization-and-quantization.
شيكاغو (المؤلف–التاريخ)
Zou, Hang, Chao Zhang, Yuzhi Yang, Yu Tian, Samson Lasaulce, and Mérouane Debbah. 2026. "FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization." https://omanscience.com/ar/articles/fedfit-federated-fine-tuning-of-llms-via-vector-bank-parameterization-and-quantization.
هارفارد
Zou, H., Zhang, C., Yang, Y., Tian, Y., Lasaulce, S. and Debbah, M. (2026) 'FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization', Available at: https://omanscience.com/ar/articles/fedfit-federated-fine-tuning-of-llms-via-vector-bank-parameterization-and-quantization.
فانكوفر
Zou H, Zhang C, Yang Y, Tian Y, Lasaulce S, Debbah M. FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization. https://omanscience.com/ar/articles/fedfit-federated-fine-tuning-of-llms-via-vector-bank-parameterization-and-quantization
IEEE
H. Zou, C. Zhang, Y. Yang, Y. Tian, S. Lasaulce, and M. Debbah, "FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization," https://omanscience.com/ar/articles/fedfit-federated-fine-tuning-of-llms-via-vector-bank-parameterization-and-quantization.