Preprint Open access
UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation
On-policy distillation from multiple teachers combines expertise from different domains in a single student, but conflicting gradients can hinder this integration. Gradient corrections directly constrain parameter updates under plain SGD. With optimizers such as AdamW, however, momentum, adaptive scaling, and weight de …