الملخص
Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Chen, J., Wang, X., Yan, T., Wu, H., Chang, X. W., Zhang, Y., & Ding, L. (2026). Sequential Functional Structured Tucker Compression for Large Language Model Attentions. https://omanscience.com/ar/articles/sequential-functional-structured-tucker-compression-for-large-language-model-attentions
MLA 9
Chen, Jiangfeng, et al. "Sequential Functional Structured Tucker Compression for Large Language Model Attentions." https://omanscience.com/ar/articles/sequential-functional-structured-tucker-compression-for-large-language-model-attentions.
شيكاغو (المؤلف–التاريخ)
Chen, Jiangfeng, Xinyu Wang, Tianshuo Yan, Hanwei Wu, Xiao-Wen Chang, Yang Zhang, and Lei Ding. 2026. "Sequential Functional Structured Tucker Compression for Large Language Model Attentions." https://omanscience.com/ar/articles/sequential-functional-structured-tucker-compression-for-large-language-model-attentions.
هارفارد
Chen, J., Wang, X., Yan, T., Wu, H., Chang, X. W., Zhang, Y. and Ding, L. (2026) 'Sequential Functional Structured Tucker Compression for Large Language Model Attentions', Available at: https://omanscience.com/ar/articles/sequential-functional-structured-tucker-compression-for-large-language-model-attentions.
فانكوفر
Chen J, Wang X, Yan T, Wu H, Chang XW, Zhang Y, et al. Sequential Functional Structured Tucker Compression for Large Language Model Attentions. https://omanscience.com/ar/articles/sequential-functional-structured-tucker-compression-for-large-language-model-attentions
IEEE
J. Chen, X. Wang, T. Yan, H. Wu, X. W. Chang, Y. Zhang, and L. Ding, "Sequential Functional Structured Tucker Compression for Large Language Model Attentions," https://omanscience.com/ar/articles/sequential-functional-structured-tucker-compression-for-large-language-model-attentions.