Abstract
Post-training tensor-network compression replaces Transformer linear layers with Tensor Train (TT) or Tree Tensor Network (TTN) operators, but standard decompositions minimize weight-space Frobenius error rather than functional error under the layer's activation distribution. We propose Activation-aware Weight Tensorization (AWT), a training-free calibration wrapper that preconditions each weight matrix with a diagonal activation-derived scale before an unchanged TT/TTN solver and deploys the result with only an input-side elementwise rescaling. Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization at 2-6 times compression: under single-operator replacement, AWT closes 12-35% of the WikiText perplexity gap to the dense baseline across the three model families and 2-6 times compression settings; while under multi-operator Llama suffix replacement it closes 27-60% across attention-group and all-seven-matrix settings. The gains also transfer to downstream HellaSwag and ARC-Challenge evaluations. We further show that diagonal preconditioning is a robustness-modularity tradeoff rather than a diagonal-covariance assumption: a dense full-covariance oracle wins its own weighted objective in 80/81 cases, yet diagonal AWT gives better held-out functional fidelity in 53/81 cases. Together, these results position AWT as a principled, modular preconditioner for improving functional fidelity in fixed TT/TTN compression pipelines without modifying the decomposition solver.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Beatini, A., Maronese, M., & Rodolà, E. (2026). Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression. https://omanscience.com/en/articles/activation-aware-weight-tensorization-a-calibration-time-preconditioner-for-tensor-network-llm-compression
MLA 9
Beatini, Alessandro, et al. "Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression." https://omanscience.com/en/articles/activation-aware-weight-tensorization-a-calibration-time-preconditioner-for-tensor-network-llm-compression.
Chicago (author–date)
Beatini, Alessandro, Marco Maronese, and Emanuele Rodolà. 2026. "Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression." https://omanscience.com/en/articles/activation-aware-weight-tensorization-a-calibration-time-preconditioner-for-tensor-network-llm-compression.
Harvard
Beatini, A., Maronese, M. and Rodolà, E. (2026) 'Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression', Available at: https://omanscience.com/en/articles/activation-aware-weight-tensorization-a-calibration-time-preconditioner-for-tensor-network-llm-compression.
Vancouver
Beatini A, Maronese M, Rodolà E. Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression. https://omanscience.com/en/articles/activation-aware-weight-tensorization-a-calibration-time-preconditioner-for-tensor-network-llm-compression
IEEE
A. Beatini, M. Maronese, and E. Rodolà, "Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression," https://omanscience.com/en/articles/activation-aware-weight-tensorization-a-calibration-time-preconditioner-for-tensor-network-llm-compression.