Home / Curated / Architectures / HyperFormer (Shared Hypernetwork Multi-Task Adapter)

HyperFormer (Shared Hypernetwork Multi-Task Adapter)

Architectures AIMech:HyperFormerSharedHypernetworkAdapter DRAFT

Aliases: HyperFormer, HyperFormer++, Shared Hypernetwork Task-Conditional Adapter

A parameter-efficient multi-task Transformer adaptation architecture that learns low-dimensional task, layer-ID, and adapter-position embeddings projected through a MLP task projector h'_I to condition shared linear hypernetworks that generate task- and layer-specific bottleneck adapter weights (U^l_tau, D^l_tau) and conditional layer normalization parameters (gamma^l_tau, beta^l_tau).

Identity Notes & Concept Scope Introduced by Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson (ACL/IJCNLP 2021, pp. 565-576, arXiv:2106.04489). Both per-layer HyperFormer (which places a task-conditioned hypernetwork in each Transformer layer) and the compact HyperFormer++ variant (which shares a single pair of adapter and LayerNorm hypernetworks across all layers and tasks conditioned on concatenated task z_tau, layer-ID l_i, and position p_j embeddings) are lumping-cohesive variants covered together in this record.

Claims & Grounded Evidence

4 Claims

Mechanism #hyperformer-conditional-adapter-and-layernorm-mechanism

HyperFormer inserts task-conditional adapter layers A^l_tau(x) = LN^l_tau(U^l_tau(GeLU(D^l_tau(x)))) + x (Eq. 3) with conditional layer normalization LN^l_tau(x^i_tau) = gamma^l_tau (o) ((x^i_tau - mu_tau)/sigma_tau) + beta^l_tau (Eq. 4) after the attention and feed-forward blocks of a frozen Transformer, generating the adapter weights (U^l_tau, D^l_tau) = (W^{U^l}, W^{D^l}) I_tau (Eq. 6) and LayerNorm parameters (gamma^l_tau, beta^l_tau) = (W^{gamma^l}, W^{beta^l}) I_tau (Eq. 7) from a learned task embedding I_tau = h_I(z_tau) (Eq. 5).

supports Source: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks · Locator: Section 2.1-2.3, Figure 1 & Eqs. 3-7, pp. 1-4; Section 4.3 Table 4, p. 7

Mechanism #hyperformer-plusplus-cross-layer-sharing-mechanism

HyperFormer++ shares the hypernetworks across all L layers and T tasks by feeding the concatenation of task embedding z_tau in R^t, layer-ID embedding l_i in R^t, and adapter position embedding p_j in R^t through a two-layer ReLU MLP projector h'_I followed by LayerNorm: I_tau = h'_I(z_tau, l_i, p_j) (Eq. 8), reducing total trainable parameters from 4 T L (2 h d + 2 h) for standard Adapters to t(T + 4 + 2 L) + 8 t e + 2 t (2 h d + 2 h) (only 0.29% parameters per task on T5-Base with 1.02x total model size, 3x fewer trainable parameters than Adapters_BASE).

supports Source: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks · Locator: Section 2.4 HyperFormer++, Eq. 8, p. 4; Section 3.1 Table 1, p. 5; Section 4.1 Parameter Efficiency, p. 7

Application #hyperformer-glue-and-fewshot-transfer-application

Evaluated on T5-Small and T5-Base across multi-task GLUE (where HyperFormer++_BASE achieves 86.48 average score with 0.29% trainable parameters per task vs. 85.47 for multi-task T5_BASE and 84.88 for single-task Adapters_BASE), low-resource GLUE subsampling (100 to 4,000 samples per task), and few-shot domain transfer across 7 datasets (SciTail, CB, TREC, BoolQ, IMDB, Yelp Polarity, and PAWS).

supports Source: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks · Locator: Section 3.1-3.3, Tables 1-2 & Figure 2, pp. 5-6

Limitation #hyperformer-fewshot-hypernet-tuning-and-reduction-factor-limitation

During few-shot domain transfer to unseen tasks, fine-tuning only the new task embedding without also updating the hypernetwork parameters achieves lower accuracy in the few-shot regime, per-layer HyperFormer without cross-layer sharing incurs substantial parameter overhead (1.54x total parameters and 6.86% per task on T5_BASE), and the adapter reduction factor r in {8, 16, 32} must be tuned depending on dataset size.

supports Source: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks · Locator: Section 2.4, p. 4; Section 3.1 Table 1, p. 5; Section 3.3 Footnote 4, p. 6; Appendix A Table 5-7, pp. 11-12

⚠️ Curated Limitations & Negative Results

Unshared per-layer HyperFormer increases total model parameters to 1.54x (6.86% per task on T5-Base) and requires 7.60 GB GPU memory unless cross-layer hypernetwork sharing (HyperFormer++) is used.
In few-shot transfer to new tasks, updating only the task embedding underperforms jointly fine-tuning both the task embedding and hypernetwork weights.
Relative to multi-task T5-Base fine-tuning, HyperFormer++_BASE still increases training time per 1,000 steps by 30.49% (7.19 min vs. 5.51 min), and optimal adapter reduction factors r vary across dataset sizes.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	HyperFormer uses shared hypernetwork neural networks to generate task-conditional adapter and layer-normalization weights inside frozen Transformer architectures.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r703-primary-pdf`	Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson	primary paper full_text	ID: https://dblp.org/search/publ/api?q=Parameter-efficient+Multi-task+Fine-tuning+for+Transformers+via+Shared+Hypernetworks&format=xml sha256:39ac7c1b3a9f…
`r703-arxiv-abs`	Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (arXiv abstract) Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson	primary paper abstract_only	ID: https://arxiv.org/abs/2106.04489 sha256:186098afcdb3…
`r703-metadata-cr`	Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (Crossref metadata) Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=Parameter-efficient+Multi-task+Fine-tuning+for+Transformers+via+Shared+Hypernetworks&format=xml sha256:656a9f67683e…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:06:00Z

Updated: 2026-10-19T13:06:00Z

Source File: data/curated/HyperFormerSharedHypernetworkAdapter.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories