Preprint Open access
iS-KV: Online Low-Rank KV Cache Compression via Block-Incremental SVD
Long chain-of-thought reasoning substantially increases KV-cache memory during autoregressive decoding, as every generated token introduces new key and value states and causes the cache to grow linearly with decoding length. Existing KV-cache compression methods typically control this growth through token eviction, but …