Home / Curated / Algorithms / Generalized LoRA

Generalized LoRA

Algorithms AIMech:GLoRA DRAFT

Aliases: GLoRA, One-for-All Generalized Low-Rank Adaptation, GLoRA Structural Re-Parameterization Supernet PEFT

A unified parameter-efficient fine-tuning framework that adapts each pretrained linear layer f(x) = W_0 x + b_0 using five configurable support tensors (A, B, C, D, E) spanning weight scaling, weight shifting, layer-wise prompt tuning, and bias scaling and shifting, selects optimal per-layer tensor structures from a weight-entangled supernet via evolutionary search, and structurally re-parameterizes all learned tensors into a single unified weight matrix W_uni and bias vector b_uni for zero-overhead inference.

Identity Notes & Concept Scope Distinct from standard LoRA (AIMech:LoRA, which tunes only an additive low-rank weight residual B without weight scaling A, layer-wise prompt vector C, or bias scale/shift tensors D and E), from DoRA (AIMech:DoraWeightDecomposedLowRankAdaptation, which normalizes column-wise weight directions and learns a magnitude vector rather than searching heterogeneous support tensor types {LoRA, vector, scalar, none} in a multi-path supernet), and from DyLoRA (AIMech:DyLoRADynamicSearchFreeLowRankAdaptation, which orders ranks within a single LoRA pair via nested dropout rather than searching across five distinct support tensor roles). Formulated by Chavan et al. (arXiv:2306.07967).

Claims & Grounded Evidence

4 Claims

Mechanism #glora-five-support-tensor-unified-formulation

GLoRA formulates the adapted linear transformation of frozen weights W_0 in R^{d_2 x d_1} and bias b_0 in R^{d_2 x 1} as f(x) = (W_0 + W_0 A + B) x + C W_0 + D b_0 + E + b_0, where each layer's trainable support tensors are independently chosen from discrete search spaces A, B in {LoRA, vector, scalar, none}, C in {LoRA, vector, none}, and D, E in {vector, scalar, none} (432 configurations per linear layer) using weight-entangled sub-tensor indexing across a multi-path supernet.

supports Source: One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning · Locator: Section 2.2 A Unified Formulation of One-for-All, Eq. 10-11 & Appendix E, p. 4, 15

Mechanism #glora-structural-reparameterization-zero-inference-cost

Because GLoRA eliminates non-linear activations between adjacent support tensor operations, all trained support tensors are absorbed post-training into a single unified weight matrix W_uni = W_0 + W_0 A + B and unified bias vector b_uni = C W_0 + D b_0 + E + b_0 so that inference executes f(x) = W_uni x + b_uni with zero additional parameters or FLOPs.

supports Source: One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning · Locator: Section 2.3 Structural Re-parameterization Design, Eq. 12-14 & Table 4, p. 5, 8

Application #glora-vision-and-llm-transfer-application

GLoRA is applied to parameter-efficient transfer learning, few-shot recognition, and out-of-domain generalization on ViT-B and Swin-B across VTAB-1K and ImageNet variants as well as instruction tuning of LLaMA-1-7B and LLaMA-2-7B on the Open LLM Leaderboard, outperforming LoRA, VPT, Adapter, and NOAH while maintaining identical inference throughput to the unadapted backbone.

supports Source: One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning · Locator: Section 3 Experiments, Tables 1-4 and Figures 2-3, p. 5-8

Limitation #glora-supernet-training-and-evolutionary-search-overhead

Training the multi-path weight-entangled supernet followed by evolutionary subnet search on a validation set requires substantially more training epochs (e.g., 5x the training epochs of single-configuration LoRA) and additional validation evaluations before the final re-parameterized model is deployed.

supports Source: One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning · Locator: Section 2.4, Appendix A, Appendix B, and Appendix D, p. 5, 14-15

⚠️ Curated Limitations & Negative Results

Requires multi-epoch supernet training (typically 5x the training epochs of standard LoRA) followed by an evolutionary search phase over validation data to identify the optimal per-layer subnet.
Restricted to purely linear support tensor transformations without intermediate non-linear bottleneck activations so that structural re-parameterization into W_uni and b_uni remains exact.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Generalized LoRA adapts pretrained vision and language neural networks via structurally re-parameterized support tensors.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r683-primary-pdf`	One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, Zhiqiang Shen	primary paper full_text	ID: https://dblp.org/rec/journals/corr/abs-2306-07967.xml sha256:0a1f68436cea…
`r683-arxiv-abs`	One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning (arXiv abstract) Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, Zhiqiang Shen	primary paper abstract_only	ID: https://arxiv.org/abs/2306.07967 sha256:c3658c586177…
`r683-metadata-cr`	One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning (Crossref metadata) Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, Zhiqiang Shen	primary paper abstract_only	ID: https://dblp.org/rec/journals/corr/abs-2306-07967.xml sha256:0c1c3be4ae27…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:46:00Z

Updated: 2026-10-19T12:46:00Z

Source File: data/curated/GLoRA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories