Abstract
Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a token's contribution with its own loss. However, each token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by language models: generation selects each token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the token's own loss does not. Understanding and controlling what a model learns from a token therefore requires decoupling its roles.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yuan, S., Lin, R., Lin, K. Q., Yu, J., Feng, L., Russell, C., & Liu, T. (2026). Decoupling Token Roles in Autoregressive Pretraining. https://omanscience.com/en/articles/decoupling-token-roles-in-autoregressive-pretraining
MLA 9
Yuan, Suqin, et al. "Decoupling Token Roles in Autoregressive Pretraining." https://omanscience.com/en/articles/decoupling-token-roles-in-autoregressive-pretraining.
Chicago (author–date)
Yuan, Suqin, Runqi Lin, Kevin Qinghong Lin, Junchi Yu, Lei Feng, Chris Russell, and Tongliang Liu. 2026. "Decoupling Token Roles in Autoregressive Pretraining." https://omanscience.com/en/articles/decoupling-token-roles-in-autoregressive-pretraining.
Harvard
Yuan, S., Lin, R., Lin, K. Q., Yu, J., Feng, L., Russell, C. and Liu, T. (2026) 'Decoupling Token Roles in Autoregressive Pretraining', Available at: https://omanscience.com/en/articles/decoupling-token-roles-in-autoregressive-pretraining.
Vancouver
Yuan S, Lin R, Lin KQ, Yu J, Feng L, Russell C, et al. Decoupling Token Roles in Autoregressive Pretraining. https://omanscience.com/en/articles/decoupling-token-roles-in-autoregressive-pretraining
IEEE
S. Yuan, R. Lin, K. Q. Lin, J. Yu, L. Feng, C. Russell, and T. Liu, "Decoupling Token Roles in Autoregressive Pretraining," https://omanscience.com/en/articles/decoupling-token-roles-in-autoregressive-pretraining.