Abstract
Efficient language models must reduce not only the cost of individual accesses to past context but also the overhead of repeatedly selecting and processing historical information across layers. We introduce the Global State Model (GSM), a causal encoder--decoder architecture that concentrates the selection and aggregation of long-range information in the encoding stage. Through multiple stages of history retrieval, the encoder progressively incorporates long-range information into representations at recent positions, forming a shared state with a fixed window size. Each decoder layer accesses this same state using queries updated from the preceding layer, preserving computational depth while avoiding repeated construction of historical key--value (KV) representations and long-range indexing. As a result, neither the decoder's per-step attention cost nor its KV cache size grows with the history length. Experiments show that GSM improves computational efficiency and reduces cache overhead while maintaining model performance and the ability to use long-range information, offering a shared-state architecture for efficient language modeling.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zheng, Y., Wen, B., Wang, X., Jiang, K., Zheng, X., Liu, C., Fu, H., Wang, J., Zhang, T., Fan, H., Li, Y., Chen, J., Wang, X., Gao, T., & Li, H. (2026). GSM: Efficient Language Modeling with Shared Global State. https://omanscience.com/en/articles/gsm-efficient-language-modeling-with-shared-global-state
MLA 9
Zheng, Yunao, et al. "GSM: Efficient Language Modeling with Shared Global State." https://omanscience.com/en/articles/gsm-efficient-language-modeling-with-shared-global-state.
Chicago (author–date)
Zheng, Yunao, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, and Han Li. 2026. "GSM: Efficient Language Modeling with Shared Global State." https://omanscience.com/en/articles/gsm-efficient-language-modeling-with-shared-global-state.
Harvard
Zheng, Y., Wen, B., Wang, X., Jiang, K., Zheng, X., Liu, C., Fu, H., Wang, J., Zhang, T., Fan, H., Li, Y., Chen, J., Wang, X., Gao, T. and Li, H. (2026) 'GSM: Efficient Language Modeling with Shared Global State', Available at: https://omanscience.com/en/articles/gsm-efficient-language-modeling-with-shared-global-state.
Vancouver
Zheng Y, Wen B, Wang X, Jiang K, Zheng X, Liu C, et al. GSM: Efficient Language Modeling with Shared Global State. https://omanscience.com/en/articles/gsm-efficient-language-modeling-with-shared-global-state
IEEE
Y. Zheng, B. Wen, X. Wang, K. Jiang, X. Zheng, C. Liu, H. Fu, J. Wang, T. Zhang, H. Fan, Y. Li, J. Chen, X. Wang, T. Gao, and H. Li, "GSM: Efficient Language Modeling with Shared Global State," https://omanscience.com/en/articles/gsm-efficient-language-modeling-with-shared-global-state.