Abstract
Vision-language models (VLMs) have demonstrated impressive capabilities but suffer from substantial computational overhead, as vision tokens dominate the input sequence. This motivates vision token compression as a key direction to alleviate the burden. However, with the emergence of hybrid architectures incorporating linear attention (\eg, Qwen3.5), prior methods designed for softmax attention struggle to generalize. Our analysis reveals that both attention- and similarity-based approaches suffer notable performance degradation, underscoring the urgent need for compression methods tailored to this regime. To this end, we propose \textbf{V-CoLA}, an efficient training-free token compression framework specifically designed for linear attention. V-CoLA introduces a novel \textit{uniqueness-aware importance criterion} for identifying critical vision tokens, coupled with an \textit{adaptive token merging strategy} that performs compression. All components are optimized at the implementation level to remain compatible with the chunk-wise parallelism of linear attention, ensuring strong practical value. Extensive experiments across multiple benchmarks demonstrate the superiority of V-CoLA: it achieves 99.5\% of the original performance with only 50.0\% of vision tokens, and over 88.0\% with as few as 12.5\%, while delivering a 1.86$\times$ to 6.15$\times$ prefill speedup.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Jiang, H., Mao, Y., Bu, T., Zhou, H., Duan, H., Jing, W., Xu, B., Chen, X., Hu, L., Yang, B., Tao, Y., & Zhang, M. (2026). V-CoLA: Vision Token Compression with Linear Attention. https://omanscience.com/en/articles/v-cola-vision-token-compression-with-linear-attention
MLA 9
Jiang, Hao, et al. "V-CoLA: Vision Token Compression with Linear Attention." https://omanscience.com/en/articles/v-cola-vision-token-compression-with-linear-attention.
Chicago (author–date)
Jiang, Hao, Yiru Mao, Tianpeng Bu, Hao Zhou, Hongtao Duan, Wang Jing, Bowen Xu, Xin Chen, Lulu Hu, Bin Yang, Yongliang Tao, and Minying Zhang. 2026. "V-CoLA: Vision Token Compression with Linear Attention." https://omanscience.com/en/articles/v-cola-vision-token-compression-with-linear-attention.
Harvard
Jiang, H., Mao, Y., Bu, T., Zhou, H., Duan, H., Jing, W., Xu, B., Chen, X., Hu, L., Yang, B., Tao, Y. and Zhang, M. (2026) 'V-CoLA: Vision Token Compression with Linear Attention', Available at: https://omanscience.com/en/articles/v-cola-vision-token-compression-with-linear-attention.
Vancouver
Jiang H, Mao Y, Bu T, Zhou H, Duan H, Jing W, et al. V-CoLA: Vision Token Compression with Linear Attention. https://omanscience.com/en/articles/v-cola-vision-token-compression-with-linear-attention
IEEE
H. Jiang, Y. Mao, T. Bu, H. Zhou, H. Duan, W. Jing, B. Xu, X. Chen, L. Hu, B. Yang, Y. Tao, and M. Zhang, "V-CoLA: Vision Token Compression with Linear Attention," https://omanscience.com/en/articles/v-cola-vision-token-compression-with-linear-attention.