نسخة أولية وصول مفتوح
What Does Privileged Information Add to On-Policy Self-Distillation?
On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE- …