Home / Curated / Architectures / RepAdapter (Structurally Re-parameterized Visual Adapter)

RepAdapter (Structurally Re-parameterized Visual Adapter)

Architectures AIMech:RepAdapter DRAFT

Aliases: RepAdapter, RepAdapter_attn, Sequentially Re-parameterized Dense-to-Sparse Visual Adapter

A parameter-efficient visual adaptation architecture that inserts a sequential linear residual bottleneck with a dense down-projection and a group-wise sparse up-projection immediately before multi-head attention and feed-forward (or convolutional) modules during training, and losslessly merges the learned adapter weights and biases into the adjacent pre-trained projection layers via sequential structural re-parameterization for zero extra inference latency or parameters.

Identity Notes & Concept Scope Introduced by Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Rongrong Ji (2023, arXiv:2302.08106). Distinct from parallel non-linear visual adapters such as AdaptFormer (Chen et al., NeurIPS 2022; AIMech Round 696) and post-FFN non-linear sequential adapters (Houlsby et al., ICML 2019; AIMech Round 677), which cannot be merged into Transformer blocks due to intervening non-linearities, and distinct from SSF (Lian et al., NeurIPS 2022; AIMech Round 692), which applies element-wise affine modulation rather than a dense-to-sparse bottleneck projection. Both full RepAdapter (placed before both MHA and FFN) and the lightweight RepAdapter_attn variant (placed only before MHA) are covered within this single record.

Claims & Grounded Evidence

4 Claims

Mechanism #repadapter-sequential-structural-reparameterization-mechanism

By removing the internal non-linear activation from the residual bottleneck adapter f(X; theta) = (X W_d + b_d) W_u + b + X, RepAdapter simplifies the module at inference time to the single affine transformation f(X; theta) = X W_ada + b_ada with W_ada = W_d W_u + W_I and b_ada = b_d W_u + b (Eq. 9), and when sequentially pre-inserted before a pre-trained linear or convolutional layer (W_0, b_0) as X'_l = MHA(f(LN(X_{l-1}); theta)) + X_{l-1} and X_l = FFN(f(LN(X'_l); theta)) + X'_l (Eq. 12), merges into W_rep = W_ada W_0 and b_rep = b_ada W_0 + b_0 (Eq. 10) for zero additional inference parameters (Delta P = 0) and zero additional inference FLOPs or GPU latency (Delta F = 0).

supports Source: Towards Efficient Visual Adaption via Structural Re-parameterization · Locator: Section 3.2.1 Sequentially Structural Re-parameterization, Figure 3 & Eqs. 8-10, p. 4; Section 3.2.2 Adapter Placement, Figure 4 & Eq. 12, p. 5; Section 4.3.3 Table 4, p. 7

Mechanism #repadapter-dense-to-sparse-group-projection-mechanism

RepAdapter formulates its upsampling projection phi_u as a k-group sparse transformation phi_u(X') = [X'_{g_0} W_{g_0}, ..., X'_{g_k} W_{g_k}] + b where each feature split X'_i in R^{n x (c/k)} is projected by W_i in R^{(c/k) x (d/k)} (Eq. 11) and converted to a dense matrix W_u in R^{c x d} via zero-padding during inference re-parameterization, saving 25% of adapter parameters when k = 2 (0.11M vs. 0.16M for RepAdapter_attn with c = 8) while outperforming both dense (k = 1) and fully sparse (both projections group-wise) variants.

supports Source: Towards Efficient Visual Adaption via Structural Re-parameterization · Locator: Section 3.2.2 Adapter Structure, Eq. 11, p. 5; Section 4.2 Implementation Details, p. 5; Section 4.3.2 Ablation Studies, Tables 2-3, pp. 6-7

Application #repadapter-vision-benchmarks-application

Validated across 27 datasets spanning image classification (VTAB-1k, where RepAdapter achieves 76.1% average accuracy with 0.22M parameters on ViT-B/16, +7.2% over full tuning while saving 25% training time, 20% GPU memory, and 94.6% storage), CLIP few-shot and domain generalization (ImageNet, -V2, -Sketch, -A, -R), ConvNeXt-B (79.0%) and Swin-B (77.4%), video classification (SSv2 and HMDB51), and semantic segmentation (SETR on ADE20K).

supports Source: Towards Efficient Visual Adaption via Structural Re-parameterization · Locator: Abstract, p. 1; Section 4.3.1 Table 1, p. 6; Section 4.3.3 Figure 5, p. 7; Section 4.3.4 Tables 5-8, pp. 7-8

Limitation #repadapter-group-count-and-post-insertion-limitation

Increasing the group count k beyond 2 (e.g., k = 4 or k = 8) or making both projections group-wise ('full sparse') degrades VTAB-1k accuracy (from 75.5% to 74.5% and 75.1%) by restricting cross-channel information exchange, and placing RepAdapter after MHA/FFN or increasing hidden dimension c beyond 8 (e.g., c = 16) reduces accuracy due to overfitting on small downstream datasets.

supports Source: Towards Efficient Visual Adaption via Structural Re-parameterization · Locator: Section 4.3.2 Ablation Studies, Table 2(a)-(d), pp. 6-7

⚠️ Curated Limitations & Negative Results

Increasing the group number k beyond 2 or using group-wise transformations in both down- and up-projections ('full sparse') restricts cross-group feature interaction and reduces accuracy.
Expanding the bottleneck hidden dimension c from 8 to 12 or 16 degrades VTAB-1k accuracy (from 75.5% to 75.1%) due to overfitting on small-scale 1,000-example datasets.
Exact structural re-parameterization into adjacent weights requires RepAdapter to be linear and sequentially pre-inserted before linear or convolutional projections rather than placed in parallel with non-linear FFN blocks.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	RepAdapter adapts pre-trained Vision Transformer and CNN neural networks via sequentially placed linear dense-to-sparse bottleneck blocks that merge into adjacent projection weights at inference.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r701-primary-pdf`	Towards Efficient Visual Adaption via Structural Re-parameterization Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, Rongrong Ji	primary paper full_text	ID: https://dblp.org/search/publ/api?q=Towards+Efficient+Visual+Adaption+via+Structural+Re-parameterization&format=xml sha256:2d2061487728…
`r701-arxiv-abs`	Towards Efficient Visual Adaption via Structural Re-parameterization (arXiv abstract) Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, Rongrong Ji	primary paper abstract_only	ID: https://arxiv.org/abs/2302.08106 sha256:ccbe40087eea…
`r701-metadata-cr`	Towards Efficient Visual Adaption via Structural Re-parameterization (Crossref metadata) Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, Rongrong Ji	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=Towards+Efficient+Visual+Adaption+via+Structural+Re-parameterization&format=xml sha256:ed82b5cb3839…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:04:00Z

Updated: 2026-10-19T13:04:00Z

Source File: data/curated/RepAdapter.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories