نسخة أولية وصول مفتوح
Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensi …