Abstract
Pretrained transformers use little of their depth to follow references in context. Thirteen base models reliably follow only 1.4-3.6 lines, and extra pretrained loops add little. A task-trained rank-8 LoRA at one early layer extends this computation with all model weights frozen. Qwen3-8B improves from 15.5% to 99% exact accuracy on 24-line chains; a longer-trained LoRA reaches 50 lines. Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight. The LoRA starts a relay: program lines pass on their chain identity through a short range of middle layers. Frozen heads read progressively further up the chain, and removing parent-line attention stops the relay. A frozen-model measurement locates the last useful intervention layer within tolerance in three of four held-out models. Task-specific LoRAs also improve MuSiQue. Default answers therefore understate the computation accessible through a tiny edit. Code and an interactive demo are available at https://lunamos.github.io/stop-thinking-too-early/
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Jin, Z., Deng, R., & Wang, J. (2026). Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It. https://omanscience.com/en/articles/transformers-stop-thinking-too-early-and-a-tiny-lora-fixes-it
MLA 9
Jin, Zehao, et al. "Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It." https://omanscience.com/en/articles/transformers-stop-thinking-too-early-and-a-tiny-lora-fixes-it.
Chicago (author–date)
Jin, Zehao, Ruixuan Deng, and Junran Wang. 2026. "Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It." https://omanscience.com/en/articles/transformers-stop-thinking-too-early-and-a-tiny-lora-fixes-it.
Harvard
Jin, Z., Deng, R. and Wang, J. (2026) 'Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It', Available at: https://omanscience.com/en/articles/transformers-stop-thinking-too-early-and-a-tiny-lora-fixes-it.
Vancouver
Jin Z, Deng R, Wang J. Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It. https://omanscience.com/en/articles/transformers-stop-thinking-too-early-and-a-tiny-lora-fixes-it
IEEE
Z. Jin, R. Deng, and J. Wang, "Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It," https://omanscience.com/en/articles/transformers-stop-thinking-too-early-and-a-tiny-lora-fixes-it.