Home / Curated / Architectures / AdapterFusion

AdapterFusion

Architectures AIMech:AdapterFusion DRAFT

Aliases: AdapterFusion Non-Destructive Task Composition, Two-Stage Adapter Knowledge Extraction and Composition, Contextual Attention Adapter Mixer

A two-stage multi-task transfer learning architecture and algorithm that first extracts task-specific knowledge into N bottleneck adapters {Phi_1, ..., Phi_N} while keeping the pretrained Transformer backbone Theta fixed, and second composes those frozen adapters for a target task m by learning layer-wise Query, Key, and Value matrices (Q_l, K_l, V_l) that compute contextual attention weights s_{l,t} = softmax(h_{l,t}^T Q_l dot z_{l,t,n}^T K_l) between the feed-forward sub-layer representation h_{l,t} and each adapter output z_{l,t,n} to form o_{l,t} = sum_{n=1}^N s_{l,t,n} (z_{l,t,n}^T V_l).

Identity Notes & Concept Scope Distinct from sequential fine-tuning (which overwrites model parameters and suffers catastrophic forgetting), from joint multi-task adapter training (MT-A, which requires simultaneous access to all datasets and suffers inter-task gradient interference), and from LoRA composition methods such as LoraHub (AIMech:LoraHub) and MoLE (AIMech:MoLEMixtureOfLoRAExperts). Formulated by Pfeiffer, Kamath, Ruckle, Cho, and Gurevych (EACL 2021, pp. 487-503, arXiv:2005.00247).

Claims & Grounded Evidence

4 Claims

Mechanism #adapterfusion-two-stage-non-destructive-learning-algorithm

AdapterFusion separates transfer learning into Stage 1 (Knowledge Extraction), which trains task-specific adapter parameters Phi_n = argmin_Phi L_n(D_n; Theta_0, Phi) independently for each task n in {1, ..., N}, and Stage 2 (Knowledge Composition), which freezes both Theta_0 and {Phi_1, ..., Phi_N} while optimizing target-task fusion parameters Psi_m = argmin_Psi L_m(D_m; Theta_0, Phi_1, ..., Phi_N, Psi), guaranteeing zero catastrophic forgetting of source-task representations.

supports Source: AdapterFusion: Non-Destructive Task Composition for Transfer Learning · Locator: Section 2.2.1 & Section 3.1 Learning algorithm, p. 3-4

Mechanism #adapterfusion-query-key-value-contextual-mixer-mechanism

At each Transformer layer l and token step t, AdapterFusion projects the feed-forward sub-layer output h_{l,t} as the query h_{l,t}^T Q_l and each adapter's output z_{l,t,n} as both key z_{l,t,n}^T K_l and value z'_{l,t,n} = z_{l,t,n}^T V_l (with V_l initialized near the identity diag(1) + N(0, 10^-6) and l2-regularized), computing o_{l,t} = s_{l,t}^T [z'_{l,t,1}, ..., z'_{l,t,N}] where s_{l,t} = softmax(h_{l,t}^T Q_l dot z_{l,t,n}^T K_l).

supports Source: AdapterFusion: Non-Destructive Task Composition for Transfer Learning · Locator: Section 3.2 Components & Section 4.1 Experimental Setup, Figure 2, p. 4-5

Application #adapterfusion-nlu-transfer-application

AdapterFusion is evaluated across 16 NLU datasets spanning commonsense reasoning, sentiment analysis, natural language inference, sentence relatedness, argument mining, and reading comprehension on BERT-base and RoBERTa-base, outperforming full fine-tuning, Single-Task Adapters (ST-A), and Multi-Task Adapters (MT-A), with gains of +6.5% on RTE and +5.64% on MRPC by dynamically attending to high-resource MNLI and QQP adapters.

supports Source: AdapterFusion: Non-Destructive Task Composition for Transfer Learning · Locator: Section 5-6, Tables 1-2 & Figures 3-4, p. 5-8

Limitation #adapterfusion-linear-adapter-pool-scaling-limitation

Because AdapterFusion evaluates all N task adapters in parallel at every layer before applying the fusion attention mixer, both training and inference computational cost scale linearly with the number of composed tasks N unless inactive adapters are pruned post-training.

supports Source: AdapterFusion: Non-Destructive Task Composition for Transfer Learning · Locator: Section 7 Contemporary Work & Section 8.2 Outlook, p. 8-9

⚠️ Curated Limitations & Negative Results

Evaluating N parallel task adapters at every Transformer layer increases forward and backward computational cost linearly with N unless inactive adapters are pruned.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	broad_match	AdapterFusion is a modular neural network composition layer that combines multiple task-specific bottleneck adapter sub-networks via contextual attention.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r690-primary-pdf`	AdapterFusion: Non-Destructive Task Composition for Transfer Learning Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckle, Kyunghyun Cho, Iryna Gurevych	primary paper full_text	ID: https://dblp.org/search/publ/api?q=2005.00247&format=xml sha256:bf3db697845f…
`r690-arxiv-abs`	AdapterFusion: Non-Destructive Task Composition for Transfer Learning (arXiv abstract) Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckle, Kyunghyun Cho, Iryna Gurevych	primary paper abstract_only	ID: https://arxiv.org/abs/2005.00247 sha256:9168694a0773…
`r690-metadata-cr`	AdapterFusion: Non-Destructive Task Composition for Transfer Learning (Crossref metadata) Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckle, Kyunghyun Cho, Iryna Gurevych	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=2005.00247&format=xml sha256:49019c80d148…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:53:00Z

Updated: 2026-10-19T12:53:00Z

Source File: data/curated/AdapterFusion.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories