Home / Curated / Algorithms / Tied-LoRA

Tied-LoRA

Algorithms AIMech:TiedLoRA DRAFT

Aliases: Tied Low-Rank Adaptation, Weight-Tied Selective Low-Rank Adaptation, Tied-LoRA Weight-Tied PEFT

A parameter-efficient fine-tuning algorithm that enhances low-rank adaptation by sharing low-rank projection matrices across transformer layers while pairing them with diagonal scaling vectors and selectively freezing or updating subsets of the projection matrices and scaling vectors.

Identity Notes & Concept Scope Distinct from standard LoRA (AIMech:LoRA, which assigns independent untied projection matrices B_i and A_i to every adapted layer without diagonal scaling vectors), from VeRA (AIMech:VeRA, which freezes untied randomly initialized matrices B_i and A_i and trains only scaling vectors, a special case termed LoRA_vbuA in the Tied-LoRA taxonomy), and from PRoLoRA (AIMech:PRoLoRA, which tiles intra-layer chunks via roll-based partial rotation and block-shared submatrices rather than tying full low-rank projection matrices across transformer layers with selective training of {v, B, u, A}). Formulated by Renduchintala, Konuk, and Kuchaiev (NAACL 2024, arXiv:2311.09578).

Claims & Grounded Evidence

4 Claims

Mechanism #tiedlora-unified-scaling-projection-formulation

Tied-LoRA parameterizes the adaptation update of the i-th pretrained weight matrix W_0^i in R^{d x k} as W^i = W_0^i + (alpha / r) Diagonal(v_i) B_i Diagonal(u_i) A_i, where A_i in R^{r x k} and B_i in R^{d x r} are low-rank projection matrices of rank r << min(d, k), u_i in R^r and v_i in R^d are scaling vectors, and each group of parameters in {v, B, u, A} can be independently tied across all L layers or kept untied and independently set as trainable or frozen.

supports Source: Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying · Locator: Section 2 Methodology, Eq. 2 and Table 1, p. 2-3

Mechanism #tiedlora-tl6-parameter-count-scaling

In the Tied-LoRA TL6 configuration (Tied-LoRA_vbua), the low-rank projection matrices A and B are shared across all L transformer layers and trained jointly with untied layer-specific scaling vectors v_i and u_i, reducing the total number of trainable parameters across L adapted linear layers of square dimension d from 2 L d r in standard LoRA to 2 d r + L d + L r, so that cross-layer rank increases scale independently of the layer count L for the shared projection matrices.

supports Source: Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying · Locator: Section 2.1 Configuration Parameter Counts, Table 1, p. 2-3

Application #tiedlora-multi-task-language-adaptation

Tied-LoRA is applied to parameter-efficient fine-tuning of large language models across extractive question answering, summarization, natural language inference, commonsense reasoning, arithmetic reasoning, and translation tasks, where the TL6 configuration retains comparable accuracy to standard LoRA across low and high ranks while requiring approximately 12.5% to 13% of the trainable parameters.

supports Source: Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying · Locator: Section 3 Experiments & Section 4 Results and Analysis, Table 2, p. 3-6

Limitation #tiedlora-dimension-matching-and-frozen-projection-degradation

Cross-layer weight tying in Tied-LoRA requires adapted weight matrices within a tied group to share identical input and output dimensions, and configurations that freeze randomly initialized tied projection matrices while training only scaling vectors degrade sharply on complex generation tasks such as arithmetic reasoning (GSM8k) and machine translation.

supports Source: Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying · Locator: Section 4 Results and Analysis & Limitations, p. 4-6

⚠️ Curated Limitations & Negative Results

Cross-layer weight tying requires the adapted linear projections within a tied group to share matching matrix dimensions.
Configurations that freeze both tied low-rank projection matrices and train only scaling vectors underperform substantially on complex reasoning and sequence generation tasks.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Tied-LoRA adapts pretrained neural network weight matrices using weight-tied low-rank projections and diagonal scaling vectors.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r677-primary-pdf`	Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying Adithya Renduchintala, Tugrul Konuk, Oleksii Kuchaiev	primary paper full_text	ID: https://dblp.org/rec/conf/naacl/RenduchintalaKK24.xml sha256:a37ba02f5d0b…
`r677-arxiv-abs`	Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying (arXiv abstract) Adithya Renduchintala, Tugrul Konuk, Oleksii Kuchaiev	primary paper abstract_only	ID: https://arxiv.org/abs/2311.09578 sha256:88d2c0574850…
`r677-metadata-cr`	Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying (Crossref metadata) Adithya Renduchintala, Tugrul Konuk, Oleksii Kuchaiev	primary paper abstract_only	ID: https://dblp.org/rec/conf/naacl/RenduchintalaKK24.xml sha256:0d39df6306c6…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:40:00Z

Updated: 2026-10-19T12:40:00Z

Source File: data/curated/TiedLoRA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories