Preprint Open access
Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
Softmax attention, at the heart of Transformers, has demonstrated remarkable capabilities. Yet its underlying mechanisms remain only partially understood. Recent theoretical work studies Gaussian prompts, where the infinite-prompt limit reduces softmax attention to a linear map, but also removes the query-dependent sel …