Preprint Open access
Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines
Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the parameter they changed rather than non-determinism in the training data sequence. The data loader must provide elastic determinism, i.e., a …