Preprint Open access
LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their ex …