Preprint Open access
Shared Stopping Decisions Change Answers in HQQ Cache Quantization
Language-model systems batch questions for throughput, but unrelated questions should not change a target's answer when its input and numerical execution are fixed. We study compression of the key and value cache, which stores attention representations reused during generation. With request-local groups, Transformers' …