AdaptFormer
Architectures AIMech:AdaptFormer DRAFT
Aliases: AdaptMLP, AdaptFormer-Swin, AdaptFormer Parallel Vision Transformer Adapter, Parallel MLP Bottleneck Vision Transformer Adaptation
A parameter-efficient Vision Transformer adaptation architecture that replaces each encoder block's feed-forward network (MLP) with a dual-branch AdaptMLP module consisting of the frozen pre-trained MLP in the left branch and a lightweight trainable down-ReLU-up bottleneck in the right branch connected in parallel via a scaling factor s (default s = 0.10). Given post-attention tokens x'_l in R^{N x d}, the right branch computes adapted features x_tilde_l = ReLU(LN(x'_l) * W_down) * W_up with W_down in R^{d x d_hat}, W_up in R^{d_hat x d} (d_hat << d), and fuses them with the frozen MLP branch via residual connection x_l = MLP(LN(x'_l)) + s * x_tilde_l + x'_l, introducing 2*d*d_hat + d_hat + d trainable parameters per layer while keeping all pre-trained weights frozen.
Identity Notes & Concept Scope
Distinct from sequential bottleneck adapters (AIMech:BottleneckAdapter; Houlsby et al., 2019), which insert adapters sequentially after MHSA and/or MLP sub-layers, and from Visual Prompt Tuning (AIMech:VisualPromptTuning), which prepends learnable tokens into the attention sequence (and suffers optimization instability when token counts scale up on video tasks or under hierarchical shifted-window attention in Swin Transformers). Also distinct from SSF (AIMech:SSFScaleAndShiftFeatures), which applies channel-wise linear scale/shift modulation rather than a parallel nonlinear ReLU bottleneck.
Claims & Grounded Evidence
3 Claims
Mechanism
#adaptformer-parallel-relu-bottleneck-and-zero-up-initialization
AdaptFormer's AdaptMLP module computes x_tilde_l = ReLU(LN(x'_l) * W_down) * W_up in parallel with the frozen MLP block via x_l = MLP(LN(x'_l)) + s * x_tilde_l + x'_l (Equations 3-4), adding 2*d*d_hat + d_hat + d parameters per layer, and initializes W_down with Kaiming Normal/Uniform while zero-initializing W_up and all adapter biases so the adapted block exactly matches the pre-trained identity function at the start of fine-tuning (Algorithm 1).
Application
#adaptformer-image-and-video-recognition-scalability
With d_hat = 64 (1.26M trainable parameters, 1.46% of ViT-Base), AdaptFormer-64 achieves 59.02% Top-1 accuracy on Something-Something v2 and 55.69% on HMDB51 with self-supervised VideoMAE pre-training—outperforming 100% full fine-tuning (53.97% and 46.41%) by +5.05% and +9.28% and VPT (43.73% and 52.67%) by +15.29% and +3.02%—while parallel insertion outperforms sequential insertion (59.02% vs. 58.17%) and top-layer placement (layers 7-12: 57.99%) outperforms bottom-layer placement (layers 1-6: 50.48%).
Limitation
#adaptformer-scaling-factor-and-in-domain-subset-limitation
AdaptFormer requires a small scaling factor s < 1 (optimal s = 0.10) on vision tasks because large scaling factors (such as s = 4 used in NLP) cause unstable fine-tuning, and when fine-tuning on ImageNet-1K using ImageNet-21K pre-trained weights (where the target dataset is a subset of the pre-training dataset), increasing the middle dimension from d_hat = 1 (82.33%) to d_hat = 64 (81.86%) degrades accuracy due to over-perturbing the pre-trained model.
⚠️ Curated Limitations & Negative Results
- Requires a small scaling factor s < 1 (default s = 0.10) for vision tasks to avoid training instability, and has only been validated on image/video classification rather than dense prediction tasks such as object detection or semantic segmentation (Section 4.5, p. 9; Section 5, p. 10).
- When the downstream dataset is a subset of the upstream pre-training dataset (e.g., ImageNet-1K from ImageNet-21K), larger bottleneck dimensions (d_hat = 64 vs. d_hat = 1) degrade accuracy by 0.47% (Appendix A.4, Table 9).
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | AdaptFormer is a parallel bottleneck neural network adaptation architecture for Vision Transformers across image and video recognition. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r696-primary-pdf
|
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=AdaptFormer+Adapting+Vision+Transformers+Chen&format=xml
sha256:262e20ceba2c…
|
r696-arxiv-abs
|
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2205.13535
sha256:706775711153…
|
r696-metadata-cr
|
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=AdaptFormer+Adapting+Vision+Transformers+Chen&format=xml
sha256:7199c1ca4f0f…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T12:59:00Z
Updated: 2026-10-19T12:59:00Z
Source File:
data/curated/AdaptFormer.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories