Preprint Open access
OptiSelect: How does the Optimizer Shape Data Curriculum?
Online data selection has demonstrated substantial efficiency gains for LLM pretraining by training on the most valuable candidates within each batch. Since a candidate's value is realized through its effective model update, principled selection should account for the optimizer step, which reshapes the raw gradient bef …