Abstract
Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Tang, H., Kang, H., Cai, H., Han, S., & Xiong, C. (2026). Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs. https://omanscience.com/en/articles/delta-matching-closing-the-final-gap-of-native-8-bit-training-for-llms
MLA 9
Tang, Haozhan, et al. "Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs." https://omanscience.com/en/articles/delta-matching-closing-the-final-gap-of-native-8-bit-training-for-llms.
Chicago (author–date)
Tang, Haozhan, Hao Kang, Han Cai, Song Han, and Chenyan Xiong. 2026. "Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs." https://omanscience.com/en/articles/delta-matching-closing-the-final-gap-of-native-8-bit-training-for-llms.
Harvard
Tang, H., Kang, H., Cai, H., Han, S. and Xiong, C. (2026) 'Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs', Available at: https://omanscience.com/en/articles/delta-matching-closing-the-final-gap-of-native-8-bit-training-for-llms.
Vancouver
Tang H, Kang H, Cai H, Han S, Xiong C. Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs. https://omanscience.com/en/articles/delta-matching-closing-the-final-gap-of-native-8-bit-training-for-llms
IEEE
H. Tang, H. Kang, H. Cai, S. Han, and C. Xiong, "Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs," https://omanscience.com/en/articles/delta-matching-closing-the-final-gap-of-native-8-bit-training-for-llms.