Preprint Open access
Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization
Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide …