Preprint Open access
Overcoming Prior Barriers: Supervised Fine-Tuning under Long-Tail Distribution
Supervised fine-tuning (SFT) adapts pretrained large language models (LLMs) to downstream tasks, but the required concepts can receive substantially different levels of pretrained support. Frequent concepts are more likely to be well learned, whereas rare concepts may remain weakly represented. We introduce a novel not …