Preprint Open access
Query Expansion and Key Specialization in Transformer Attention Geometry
The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attention mechanisms. The question of whether there is an effect from their functional distinction on their geometric development in training rema …