Intrinsic fine-tuning subspaces are task-specific with only partial, scale-dependent transfer — a unified cross-task subspace is feasible, but no whole-parameter-space global subspace is demonstrated

intrinsic-dimensiondimensionalityfine-tuninglarge-language-models

Zhong Zhang, Bang Liu & Junming Shao, "Fine-tuning Happens in Tiny Subspaces" (arXiv:2305.17446, 2023), extend Aghajanyan et al.'s intrinsic-dimension result by extracting the actual (non-random) subspace each fine-tuning trajectory occupies via SVD of the trajectory itself, rather than testing a random projection as in the 2018 objective-landscape method. They find these subspaces are task-specific with only partial transfer: models fine-tuned in a transferred subspace still beat the random-subspace baseline ("which suggests the transferability of intrinsic task-specific subspaces"), but "the transferability of subspaces seems to correlate with the scale of the transferred task" — bigger, more complex source tasks transfer worse.

Two separate results have to be kept apart, because the paper's word "global" does not mean what a cross-task reading assumes. In §4.4 the authors do build a unified cross-task subspace — stacking the fine-tuning trajectories of all eight GLUE tasks and taking the SVD yields an 8-dimensional subspace in which "the models can be effectively fine-tuned" — and they conclude "a unified intrinsic task subspace is feasible and it contains disentangled knowledge," with the cosine similarities between different tasks' parameter vectors inside it "significantly low" (tasks occupy near-orthogonal directions of one shared subspace). The zero-shot variant, which excludes the target task's own checkpoint, "decreases significantly, but still outperforms the random baseline." Separately, in Limitations, "global" means whole-parameter-space as against the layer-wise re-parameterization they adopted from Aghajanyan et al. "in order to alleviate memory and computational burdens": "such a setting restricts us to only identifying local subspaces, rather than discovering global subspaces within the entire parameter space of a pre-trained language model. The existence of a task-specific global subspace is yet to be ascertained." That disclaimer is about the scope of their search, not about whether tasks share a subspace.

This is a definitional correction to a premise the vault's within-manifold analogy quietly assumes: that there is a single "the low-intrinsic-dimension subspace fine-tuning already lives in" that a jailbreak could move within or outside of. What the primary literature measures is a family of task-conditioned low-dimensional subspaces with partial, scale-dependent transfer — each a real, measurable object of the kind Sadtler's manifold is. The correction is narrower than "there is no shared subspace," which this paper in fact refutes: a subspace shared across eight NLU tasks is demonstrated, but the tasks sit in near-orthogonal directions within it, and nothing here establishes that the safety-relevant direction is one of those already occupied. It predates Qi et al. (2023-10-05) by roughly four months and does not test whether a jailbreak subspace overlaps ordinary-task subspaces. Sharpens what "the subspace" would even mean in question-low-dimensional-subspace-one-object-or-analogy; sits alongside claim-ponkshe-2025-safety-subspaces-not-linearly-distinct-entangled-with-general-learning and claim-springer-2026-finetuning-orthogonality-unstable-curvature-steers-into-alignment-subspace as the closest bearing evidence, per observation-no-primary-source-links-qi-2023-jailbreak-to-sadtler-2014-within-manifold-asymmetry. (Note: this Zhang is Zhong Zhang; distinct from the Zhang of claim-zhang-2026-safety-alignment-low-rank-subspace-regardless-of-model-size.)