[
    {
        "id": "osp-21421",
        "type": "article-journal",
        "title": "ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs",
        "author": [
            {
                "family": "Yin",
                "given": "Hang"
            },
            {
                "family": "Wang",
                "given": "Haozhe"
            },
            {
                "family": "Luo",
                "given": "Yuhua"
            },
            {
                "family": "Pan",
                "given": "Zhangqi"
            },
            {
                "family": "Wang",
                "given": "Xiaoxing"
            },
            {
                "family": "Yan",
                "given": "Junchi"
            }
        ],
        "URL": "https://omanscience.com/ar/articles/chainlora-geometry-preserving-task-vector-merging-for-continual-learning-in-llms",
        "language": "en",
        "issued": {
            "date-parts": [
                [
                    2026
                ]
            ]
        },
        "abstract": "Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present \\textbf{ChainLoRA}, a replay-free continual merging framework built on chain-updated task-vector geometry. From a parameter-merging perspective, we formulate a geometric view of forgetting through a measurable interaction between task updates, separating directional overlap from coefficient coupling. Building on this view, ChainLoRA combines chain-updated training with post-stream adaptive SVD merging. During training, initialization and a one-sided orthogonality proxy use only the last carrier, keeping their historical-state footprint and regularization overhead constant as the task stream grows. At merging time, Adaptive SVD extracts a shared carrier and aligns it to the latest task through Procrustes adaptation. Our theoretical analysis shows that Procrustes adaptation facilitates geometric approximate separation of shared and task-specific components. The one-sided proxy further bounds inter-task interference. An effective-rank penalty additionally promotes efficient utilization of the task subspace during continual learning. Experiments show that ChainLoRA achieves state-of-the-art performance among the evaluated replay-free methods on the Large and SuperNI benchmarks, while remaining competitive on Standard CL and attaining almost the closest average scores to the evaluated replay-based method across all three benchmarks."
    }
]