Preprint Open access
Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers
Deterministic inference is essential for reliable and trustworthy machine learning. Prior studies of text generation have shown that changing factors such as batch size, batch composition, hardware, or inference engine can alter the generated text, even when the prompt, model parameters, and sampling randomness are fix …