Abstract
Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Lodhi, M. A., Zhou, C., & Burkholz, R. (2026). RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning. https://omanscience.com/en/articles/rosa-rotational-sparse-adaptation-for-memory-efficient-fine-tuning
MLA 9
Lodhi, Muhammad Azeem, et al. "RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning." https://omanscience.com/en/articles/rosa-rotational-sparse-adaptation-for-memory-efficient-fine-tuning.
Chicago (author–date)
Lodhi, Muhammad Azeem, Chao Zhou, and Rebekka Burkholz. 2026. "RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning." https://omanscience.com/en/articles/rosa-rotational-sparse-adaptation-for-memory-efficient-fine-tuning.
Harvard
Lodhi, M. A., Zhou, C. and Burkholz, R. (2026) 'RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning', Available at: https://omanscience.com/en/articles/rosa-rotational-sparse-adaptation-for-memory-efficient-fine-tuning.
Vancouver
Lodhi MA, Zhou C, Burkholz R. RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning. https://omanscience.com/en/articles/rosa-rotational-sparse-adaptation-for-memory-efficient-fine-tuning
IEEE
M. A. Lodhi, C. Zhou, and R. Burkholz, "RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning," https://omanscience.com/en/articles/rosa-rotational-sparse-adaptation-for-memory-efficient-fine-tuning.