Abstract
Model merging aims to consolidate multiple task-specific models without access to extra training process. However, existing subspace-based methods largely rely on a uniform treatment of task updates, overlooking their intrinsic spectral and depth-wise heterogeneity. We identify two key deviations from this assumption: different tasks require different subspace capacity and exhibit different tolerance to spectral transformation, while subspace projection introduces depth-dependent distortion. Based on these observations, we propose SADA-Merging, a spectrum-aware and depth-adaptive framework for data-free model merging. SADA-Merging allocates task-specific subspace capacity according to spectral complexity, adapts spectral preservation according to task-wise plasticity, and applies depth-dependent anchoring to compensate for projection-induced distortion. This enables the fusion process to adapt to both the intrinsic geometry of each task and its sensitivity across network depth. SADA-Merging operates directly on task updates and is applicable to both full fine-tuning and LoRA settings. Extensive experiments demonstrate consistent improvements over existing data-free merging methods across different task scales and adaptation settings.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Gu, R., Wang, Z., & Wang, W. (2026). Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging. https://omanscience.com/en/articles/beyond-uniform-subspaces-spectrum-aware-and-depth-adaptive-fusion-for-multi-task-model-merging
MLA 9
Gu, Ruxi, et al. "Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging." https://omanscience.com/en/articles/beyond-uniform-subspaces-spectrum-aware-and-depth-adaptive-fusion-for-multi-task-model-merging.
Chicago (author–date)
Gu, Ruxi, Zilei Wang, and Wei Wang. 2026. "Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging." https://omanscience.com/en/articles/beyond-uniform-subspaces-spectrum-aware-and-depth-adaptive-fusion-for-multi-task-model-merging.
Harvard
Gu, R., Wang, Z. and Wang, W. (2026) 'Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging', Available at: https://omanscience.com/en/articles/beyond-uniform-subspaces-spectrum-aware-and-depth-adaptive-fusion-for-multi-task-model-merging.
Vancouver
Gu R, Wang Z, Wang W. Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging. https://omanscience.com/en/articles/beyond-uniform-subspaces-spectrum-aware-and-depth-adaptive-fusion-for-multi-task-model-merging
IEEE
R. Gu, Z. Wang, and W. Wang, "Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging," https://omanscience.com/en/articles/beyond-uniform-subspaces-spectrum-aware-and-depth-adaptive-fusion-for-multi-task-model-merging.