Preprint Open access
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of abstract procedurally generated data can help acquire generic inductive structure at low cost. However, this adds a pretraining stage that mu …