Preprint Open access
1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry …