الملخص
A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Hayase, T., & Karakida, R. (2026). Gaussian Equivalence for Multi-Head Self-Attention. https://omanscience.com/ar/articles/gaussian-equivalence-for-multi-head-self-attention
MLA 9
Hayase, Tomohiro, and Ryo Karakida. "Gaussian Equivalence for Multi-Head Self-Attention." https://omanscience.com/ar/articles/gaussian-equivalence-for-multi-head-self-attention.
شيكاغو (المؤلف–التاريخ)
Hayase, Tomohiro, and Ryo Karakida. 2026. "Gaussian Equivalence for Multi-Head Self-Attention." https://omanscience.com/ar/articles/gaussian-equivalence-for-multi-head-self-attention.
هارفارد
Hayase, T. and Karakida, R. (2026) 'Gaussian Equivalence for Multi-Head Self-Attention', Available at: https://omanscience.com/ar/articles/gaussian-equivalence-for-multi-head-self-attention.
فانكوفر
Hayase T, Karakida R. Gaussian Equivalence for Multi-Head Self-Attention. https://omanscience.com/ar/articles/gaussian-equivalence-for-multi-head-self-attention
IEEE
T. Hayase, and R. Karakida, "Gaussian Equivalence for Multi-Head Self-Attention," https://omanscience.com/ar/articles/gaussian-equivalence-for-multi-head-self-attention.