Home / Curated / Algorithms / MiLoRA

MiLoRA

Algorithms AIMech:MiLoRA DRAFT

Aliases: Minor Singular Component Based Low-Rank Adaptation, Minor Singular Component LoRA, Minor-Spectrum SVD Low-Rank Adaptation

A parameter-efficient low-rank adaptation algorithm that decomposes each pre-trained weight matrix via SVD into a frozen principal matrix W_p spanning the largest m - r singular values and a trainable minor matrix W_m = B_m A_m initialized from the smallest r singular values and vectors B_m = U_m sqrt(Sigma_m), A_m = sqrt(Sigma_m) V_m^T.

Identity Notes & Concept Scope Directly complementary to AIMech:PissaPrincipalSingularValuesAndSingularVectorsAdaptation (PiSSA, which initializes the trainable adapter from the top-r principal singular components and freezes the minor residual) and distinct from AIMech:LowRankAdaptation (which initializes A randomly and B to zero in an unguided subspace). MiLoRA (Minor singular component based Low-Rank Adaptation; Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, & Yun Chen, NAACL 2025, arXiv:2406.09044) partitions the SVD W = U Sigma V^T = sum_{i=1}^{m-r} sigma_i u_i v_i^T + sum_{i=m-r+1}^m sigma_i u_i v_i^T = W_p + W_m (Equations 3-5). By freezing W_p and initializing B_m = U_m sqrt(Sigma_m) and A_m = sqrt(Sigma_m) V_m^T, MiLoRA (1) satisfies W = W_p + B_m A_m exactly at initialization without setting B_m = 0 or tuning a separate alpha/r scaling factor, and (2) initializes the trainable adapter in a subspace orthogonal to the principal knowledge subspace (U_p^T B_m = 0, A_m V_p = 0), adapting long-tail/noisy singular directions to the downstream task while minimizing catastrophic forgetting of pre-trained world knowledge.

Claims & Grounded Evidence

4 Claims

Mechanism #milora-minor-singular-subspace-initialization-and-principal-orthogonality

Splitting W = U_p Sigma_p V_p^T + U_m Sigma_m V_m^T into the top-(m-r) principal components W_p (frozen) and bottom-r minor components W_m initialized as B_m = U_m sqrt(Sigma_m) and A_m = sqrt(Sigma_m) V_m^T preserves W = W_p + B_m A_m exactly at step 0 while initializing the adapter in a subspace strictly orthogonal to U_p and V_p without requiring an alpha/r scaling hyperparameter.

supports Source: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning · Locator: Section 3 (Methodology, Equations 3-5 & Figure 1), pp. 2-3

Mechanism #milora-reduced-principal-feature-amplification-and-lower-forgetting-loss

Subspace similarity and projection analyses on LLaMA2-7B show that MiLoRA updates concentrate in the bottom singular vector subspace of W with a much smaller principal-feature projection ||U^T W V||_F than PiSSA (1.86 vs. 17.57) and achieve lower pre-trained distribution forgetting cross-entropy on WikiText-103 (2.54 vs. 3.24 for LoRA and 6.07 for PiSSA).

supports Source: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning · Locator: Section 5.3-5.4 (Tables 7-8, Figure 2) & Appendix B.2 (Table 14), pp. 7-8, 14

Application #milora-application-commonsense-math-instruction-and-multimodal-finetuning

Across 18 benchmarks on LLaMA2-7B, LLaMA3-8B, Qwen2.5-7B, and LLaVA1.5-7B, MiLoRA outperforms both LoRA and PiSSA on commonsense reasoning (+1.6/+5.4 on LLaMA2-7B, +1.1/+6.5 on LLaMA3-8B), math reasoning (+2.0/+3.7 EM on GSM8K/MATH), instruction following (+2.9/+2.7 on AlpacaEval 2.0/FollowBench/IFEval), and visual instruction tuning (+1.4/+4.4 on 7 VQA benchmarks) with zero inference overhead.

supports Source: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning · Locator: Section 4.1-4.2 (Tables 1-4), Section 5.1 (Table 5) & Appendix B.1 (Table 13), pp. 4-6, 13-14

Limitation #milora-limitation-soft-orthogonality-and-small-lr-large-batch-regime

Because MiLoRA enforces orthogonality between W_m and W_p only at initialization rather than via hard projection constraints during training, and because minor singular values are small in magnitude, MiLoRA underperforms PiSSA when trained for 1 epoch under a small learning rate (2e-5) and large batch size (128) regime (22.6 vs. 29.4 average on GSM8K/MATH).

supports Source: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning · Locator: Section 3 (p. 3) & Section 5.5 (Comparison Between MiLoRA and PiSSA, Table 9, p. 8)

⚠️ Curated Limitations & Negative Results

Guides optimization toward the minor singular subspace only through initialization rather than enforcing strict orthogonality constraints throughout fine-tuning.
Requires an appropriate PEFT learning rate (e.g., 3e-4); under small-learning-rate (2e-5) single-epoch schedules tailored for full fine-tuning or PiSSA, the small initial singular values of W_m slow adaptation.
Requires a one-time per-layer SVD factorization of pre-trained weight matrices before fine-tuning.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	MiLoRA adapts linear projection matrices in pre-trained transformer neural networks by fine-tuning their minor SVD components.
`AIO:PrincipalComponentAnalysis` Principal Component Analysis	related_match	MiLoRA relies on singular value decomposition to separate principal singular components from minor singular components of weight matrices.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r675-primary-pdf`	MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen	primary paper full_text	ID: https://dblp.org/search/publ/api?q=MiLoRA+Harnessing+Minor+Singular+Components&format=xml sha256:82f52cb6080e…
`r675-arxiv-abs`	MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (arXiv abstract) Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen	primary paper abstract_only	ID: https://arxiv.org/abs/2406.09044 sha256:31219bdbdda3…
`r675-metadata-cr`	MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (Crossref metadata) Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=MiLoRA+Harnessing+Minor+Singular+Components&format=xml sha256:bd20e09a3352…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:38:00Z

Updated: 2026-10-19T12:38:00Z

Source File: data/curated/MiLoRA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories