نسخة أولية وصول مفتوح
Latent-MOPD: Latent Multi-Teacher On-Policy Distillation
On-policy distillation (OPD) trains a student on the responses it generates. Existing LLM multi-teacher OPD transfers what specialists predict through their output distributions. We introduce Latent-MOPD, to our knowledge the first representation-level multi-teacher OPD method for LLMs. It integrates existing specialis …