الملخص
Low-rank compression reduces the cost of pretrained language models by replacing linear transformations with low-rank factorizations. However, conventional methods use a fixed rank allocation during inference, assigning the same amount of compute regardless of the input token. We introduce Low-Rank Conditional Computation (LRCC), which adds token-dependent computation to pretrained models by training one lightweight router per Transformer block to select among a small set of nested low-rank paths. During training, the low-rank factors remain frozen, and only the routers are optimized. We evaluate LRCC on Llama and Qwen models for language modeling and zero-shot downstream tasks. Within the same average active-parameter budget, LRCC improves the predictive performance over static low-rank compression, including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B over static methods. At matched batch-size-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama-3.2-1B and remains competitive on Llama-2-7B, without specialized kernels. Finally, we assess the usefulness of assigning a token-wise path by analyzing the routers' path choices.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Fontanari, T. V., Rulli, M. E., Alvetreti, F., Genovese, D., & Scardapane, S. (2026). LRCC: Generalizing Low-Rank Compression with Conditional Computation. https://omanscience.com/ar/articles/lrcc-generalizing-low-rank-compression-with-conditional-computation
MLA 9
Fontanari, Thomas Vaitses, et al. "LRCC: Generalizing Low-Rank Compression with Conditional Computation." https://omanscience.com/ar/articles/lrcc-generalizing-low-rank-compression-with-conditional-computation.
شيكاغو (المؤلف–التاريخ)
Fontanari, Thomas Vaitses, Maximo Eduardo Rulli, Federico Alvetreti, Donatella Genovese, and Simone Scardapane. 2026. "LRCC: Generalizing Low-Rank Compression with Conditional Computation." https://omanscience.com/ar/articles/lrcc-generalizing-low-rank-compression-with-conditional-computation.
هارفارد
Fontanari, T. V., Rulli, M. E., Alvetreti, F., Genovese, D. and Scardapane, S. (2026) 'LRCC: Generalizing Low-Rank Compression with Conditional Computation', Available at: https://omanscience.com/ar/articles/lrcc-generalizing-low-rank-compression-with-conditional-computation.
فانكوفر
Fontanari TV, Rulli ME, Alvetreti F, Genovese D, Scardapane S. LRCC: Generalizing Low-Rank Compression with Conditional Computation. https://omanscience.com/ar/articles/lrcc-generalizing-low-rank-compression-with-conditional-computation
IEEE
T. V. Fontanari, M. E. Rulli, F. Alvetreti, D. Genovese, and S. Scardapane, "LRCC: Generalizing Low-Rank Compression with Conditional Computation," https://omanscience.com/ar/articles/lrcc-generalizing-low-rank-compression-with-conditional-computation.