Abstract
Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yang, S. H., Huang, S. C., Lin, C. Y., Chen, Y. N., Sun, S. H., & Lee, H. Y. (2026). A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic. https://omanscience.com/en/articles/a-broader-look-at-model-merging-rethinking-implicit-regularization-induced-by-task-arithmetic
MLA 9
Yang, Sin-Han, et al. "A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic." https://omanscience.com/en/articles/a-broader-look-at-model-merging-rethinking-implicit-regularization-induced-by-task-arithmetic.
Chicago (author–date)
Yang, Sin-Han, Shih-Cheng Huang, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, and Hung-yi Lee. 2026. "A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic." https://omanscience.com/en/articles/a-broader-look-at-model-merging-rethinking-implicit-regularization-induced-by-task-arithmetic.
Harvard
Yang, S. H., Huang, S. C., Lin, C. Y., Chen, Y. N., Sun, S. H. and Lee, H. Y. (2026) 'A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic', Available at: https://omanscience.com/en/articles/a-broader-look-at-model-merging-rethinking-implicit-regularization-induced-by-task-arithmetic.
Vancouver
Yang SH, Huang SC, Lin CY, Chen YN, Sun SH, Lee HY. A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic. https://omanscience.com/en/articles/a-broader-look-at-model-merging-rethinking-implicit-regularization-induced-by-task-arithmetic
IEEE
S. H. Yang, S. C. Huang, C. Y. Lin, Y. N. Chen, S. H. Sun, and H. Y. Lee, "A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic," https://omanscience.com/en/articles/a-broader-look-at-model-merging-rethinking-implicit-regularization-induced-by-task-arithmetic.