Preprint Open access
Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, tr …