Home / Curated / Algorithms / Compacter (Low-Rank Hypercomplex Adapter Layers)

Compacter (Low-Rank Hypercomplex Adapter Layers)

Algorithms AIMech:CompacterLowRankHypercomplexAdapterLayers DRAFT

Aliases: Compacter, Compacter++, Low-Rank Parameterized Hypercomplex Multiplication Adapter, LPHM Adapter

Compacter (Compact Adapter) is a parameter-efficient fine-tuning method for pretrained Transformer models that replaces the dense down-projection D^l in R^{k x d} and up-projection U^l in R^{d x k} of bottleneck adapter modules A^l(x) = U^l(GeLU(D^l(x))) + x with Low-rank Parameterized Hypercomplex Multiplication (LPHM) layers: A^l(x) = LPHM^{U^l}(GeLU(LPHM^{D^l}(x))) + x. For a user-specified hyperparameter n dividing input dimension k and bottleneck dimension d, each LPHM weight matrix W in R^{k x d} is synthesized as the sum of n Kronecker products between shared "slow" weights A_i in R^{n x n} (shared across all adapter layers and positions in the network) and adapter-specific "fast" rank-r matrices B_i = s_i t_i^T in R^{(k/n) x (d/n)} with s_i in R^{(k/n) x r} and t_i in R^{r x (d/n)} (typically r = 1): W = sum_{i=1}^n A_i (x) (s_i t_i^T). Its variant Compacter++ retains only the post-feed-forward Compacter module per Transformer block.

Identity Notes & Concept Scope Distinct from standard bottleneck adapters (Houlsby et al., 2019), which parameterize U^l and D^l as unconstrained dense matrices costing 4 L k d parameters across L encoder-decoder layers, and from PHM-Adapter, which does not share A_i across layers or factorize B_i into rank-one outer products (costing 4 L (k d / n + n^3) parameters). Also distinct from LoRA (Hu et al., 2022), which adds parallel low-rank updates directly to frozen attention weights without a nonlinear GeLU bottleneck or Kronecker hypercomplex multiplication.

Claims & Grounded Evidence

3 Claims

Mechanism #r659-claim-linear-sum-parameter-complexity-reduction

By sharing the n matrices {A_i}_{i=1}^n in R^{n x n} (n^3 parameters total) across all layers and parameterizing each down- and up-projection's B_i via rank-one vectors s_i in R^{k/n} and t_i in R^{d/n} (n(k/n + d/n) = k + d parameters per projection), Compacter reduces total adapter weight parameters for an L-layer encoder-decoder Transformer from 4 L k d (O(k d)) in standard adapters and 4 L (k d / n + n^3) (O(k d / n)) in PHM-Adapters to 4 L (k + d) + n^3 (O(k + d)), and to 2 L (k + d) + n^3 in Compacter++.

supports Source: Compacter: Efficient Low-Rank Hypercomplex Adapter Layers · Locator: Section 4 (Parameter Efficiency), p. 5; Appendix B, Table 6, p. 16

Application #r659-claim-glue-superglue-and-low-resource-generalization

On T5-base (222M parameters), Compacter (0.073% trainable parameters) and Compacter++ (0.047% trainable parameters, 2127.66x fewer than full fine-tuning) match or outperform full fine-tuning and prior PEFT baselines (Adapter, Pfeiffer-Adapter, AdapterDrop, Prompt Tuning, Intrinsic-SAID, BitFit) on GLUE (86.62 and 86.47 vs. 86.50 average) and SuperGLUE (71.57 and 71.82 vs. 70.06 average) as well as low-resource GLUE subsamples of 100 to 4000 examples while reducing peak GPU training memory by 41.94% and epoch time by 26.51%.

supports Source: Compacter: Efficient Low-Rank Hypercomplex Adapter Layers · Locator: Section 5.3-5.6, Tables 1-3 & Figure 4, p. 7-10

Limitation #r659-claim-nonlinear-bottleneck-unmergeable-inference-limitation

Because Compacter inserts serial adapter blocks A^l(x) = LPHM^{U^l}(GeLU(LPHM^{D^l}(x))) + x containing a nonlinear GeLU activation after attention and feed-forward sublayers, its learned weights cannot be algebraically folded into the frozen backbone weight matrices at inference time, and on small backbones such as T5-small (60M) all parameter-efficient methods including Compacter underperform full fine-tuning.

supports Source: Compacter: Efficient Low-Rank Hypercomplex Adapter Layers · Locator: Section 2.2 & 3.2, Equation (2) & p. 5; Appendix E (Impact of Model Size), Table 9, p. 18

⚠️ Curated Limitations & Negative Results

Requires input dimension k and adapter bottleneck dimension d to be divisible by the Kronecker hyperparameter n, and because of the intermediate GeLU nonlinearity, the adapter layers cannot be merged into the frozen Transformer linear projections at inference time.
Trainable LayerNorm parameters and LPHM bias vectors account for over 50% of the remaining trainable parameters (e.g., 44.01% LayerNorm and 18.15% biases in Compacter++ at n = 4), so increasing n beyond 4 yields diminishing parameter savings unless biases and LayerNorm updates are omitted.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Compacter inserts Kronecker-factored LPHM bottleneck adapter layers into pretrained Transformer neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r659-primary-pdf`	Compacter: Efficient Low-Rank Hypercomplex Adapter Layers Rabeeh Karimi Mahabadi, James Henderson, Sebastian Ruder	primary paper full_text	ID: https://dblp.org/rec/conf/nips/MahabadiHR21.html sha256:ef87e29bf8fa…
`r659-arxiv-abs`	Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (arXiv abstract) Rabeeh Karimi Mahabadi, James Henderson, Sebastian Ruder	primary paper abstract_only	ID: https://arxiv.org/abs/2106.04647 sha256:7588d8592cb3…
`r659-metadata-cr`	Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (Crossref metadata) Rabeeh Karimi Mahabadi, James Henderson, Sebastian Ruder	primary paper abstract_only	ID: https://dblp.org/rec/conf/nips/MahabadiHR21.html sha256:339bb4474dad…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:22:00Z

Updated: 2026-10-19T12:22:00Z

Source File: data/curated/CompacterLowRankHypercomplexAdapterLayers.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories