Home / Curated / Architectures / Ladder Side-Tuning (LST)

Ladder Side-Tuning (LST)

Architectures AIMech:LadderSideTuning DRAFT

Aliases: LST, Ladder Side-Tuning, Backpropagation-Free Gated Ladder Side Network

A parameter- and memory-efficient transfer learning architecture that separates all trainable parameters from a frozen backbone network f into a parallel lightweight side network g of hidden dimension scaled by 1/r, fuses downsampled intermediate backbone activations h_i^f with side-network representations h_{i-1}^g via learned temperature-scaled sigmoid gates mu_i * h_i^f + (1 - mu_i) * h_{i-1}^g (mu_i = sigmoid(alpha_i / T)), initializes g via Fisher-information or magnitude structural pruning of f, and completely eliminates backpropagation through the frozen backbone.

Identity Notes & Concept Scope Distinct from in-backbone PETL methods such as LoRA (AIMech:LoRA), LoRA-FA (AIMech:LoRAFA), Compacter (AIMech:CompacterLowRankHypercomplexAdapterLayers), and IA3 (AIMech:IA3ActivationRescaling), all of which insert trainable parameters inside the backbone and must cache backbone nonlinearity derivatives {sigma'} for chain-rule backpropagation (capping memory savings at <= 50%), and from vanilla Side-Tuning (which fuses representations only at the final output layer without intermediate gated ladder shortcuts, structural pruning initialization, or side-network layer dropping). Formulated by Sung, Cho, and Bansal (NeurIPS 2022, arXiv:2206.06522).

Claims & Grounded Evidence

4 Claims

Mechanism #lst-backbone-backpropagation-elimination-memory-mechanism

Because in-backbone PETL methods must compute dL/da_{i+1} = (dL/da_{i+2}) sigma'_{i+1} W_{i+1} through the frozen backbone, they cannot reduce the nonlinearity derivative cache |sigma'| (limiting backpropagation memory savings to at most 50%), whereas LST detaches trainable parameters into a separate side network g with dimension reduction factor r so that backpropagation memory scales as (|a| + |sigma'|) / r.

supports Source: LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning · Locator: Section 3.1-3.2, Eq. 1-2 and Figure 2-3, p. 2, 4-5

Mechanism #lst-gated-ladder-connections-and-structural-pruning-init

At each side-network layer i, LST downsamples the intermediate backbone activation by 1/r and combines it with the previous side-layer activation h_{i-1}^g via mu_i * h_i^f + (1 - mu_i) * h_{i-1}^g with mu_i = sigmoid(alpha_i / T) (T = 0.1), initializes side-network weights by selecting the top d_out / r rows and corresponding columns of backbone matrices ranked by row-wise empirical Fisher information I = (1/N) sum (nabla_W log p(y_i | x_i))^2 or weight magnitude, and optionally drops intermediate side-network layers.

supports Source: LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning · Locator: Section 3.2-3.4, Figure 3-4, p. 5-7

Application #lst-glue-and-vision-language-transfer-application

LST is applied to T5-base, T5-large, and T5-3B on the GLUE benchmark and to CLIP-T5 on vision-and-language tasks (VQA, GQA, NLVR2, MSCOCO), saving 69% of full fine-tuning GPU training memory (2.7x more memory savings than Adapter and LoRA at matched parameter budgets) and enabling T5-large/T5-3B and CLIP-T5 adaptation within a 16GB GPU budget.

supports Source: LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning · Locator: Section 5 Experimental Results, Tables 1-3 and Figures 1, 5-6, p. 2, 8-9

Limitation #lst-sequential-ladder-inference-dependency-limitation

Unlike LoRA modules that merge into backbone weights W_0 with zero extra inference graph nodes, LST retains the separate side network g and linear downsample/upsample ladder projections at inference time, requiring parallel execution of backbone and side-network layers to avoid extra inference latency.

supports Source: LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning · Locator: Section 1, p. 2 & Section 5 Table 1 and Table 3 Inference Memory columns, p. 8-9

⚠️ Curated Limitations & Negative Results

Cannot be algebraically merged into the frozen backbone weights at inference time and requires executing the side network alongside the backbone.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	broad_match	Ladder Side-Tuning (LST) is a parallel side-network neural architecture connected to a frozen backbone via gated shortcut projections.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r691-primary-pdf`	LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning Yi-Lin Sung, Jaemin Cho, Mohit Bansal	primary paper full_text	ID: https://dblp.org/search/publ/api?q=2206.06522&format=xml sha256:ac80e04a0204…
`r691-arxiv-abs`	LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (arXiv abstract) Yi-Lin Sung, Jaemin Cho, Mohit Bansal	primary paper abstract_only	ID: https://arxiv.org/abs/2206.06522 sha256:10781be7c819…
`r691-metadata-cr`	LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (Crossref metadata) Yi-Lin Sung, Jaemin Cho, Mohit Bansal	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=2206.06522&format=xml sha256:5d58a3930f11…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:54:00Z

Updated: 2026-10-19T12:54:00Z

Source File: data/curated/LadderSideTuning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories