Abstract
Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83$\times$ inference speedup across diverse Looped Transformers.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Cho, S., Cui, L., Kim, S., Han, D., & Han, I. (2026). LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers. https://omanscience.com/en/articles/loopspec-pipelined-self-speculative-decoding-for-looped-transformers
MLA 9
Cho, SangLyul, et al. "LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers." https://omanscience.com/en/articles/loopspec-pipelined-self-speculative-decoding-for-looped-transformers.
Chicago (author–date)
Cho, SangLyul, Langqing Cui, Sehoon Kim, Dongsu Han, and Insu Han. 2026. "LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers." https://omanscience.com/en/articles/loopspec-pipelined-self-speculative-decoding-for-looped-transformers.
Harvard
Cho, S., Cui, L., Kim, S., Han, D. and Han, I. (2026) 'LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers', Available at: https://omanscience.com/en/articles/loopspec-pipelined-self-speculative-decoding-for-looped-transformers.
Vancouver
Cho S, Cui L, Kim S, Han D, Han I. LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers. https://omanscience.com/en/articles/loopspec-pipelined-self-speculative-decoding-for-looped-transformers
IEEE
S. Cho, L. Cui, S. Kim, D. Han, and I. Han, "LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers," https://omanscience.com/en/articles/loopspec-pipelined-self-speculative-decoding-for-looped-transformers.