Home / Curated / Architectures / DynaBERT: Dynamic BERT with Adaptive Width and Depth

DynaBERT: Dynamic BERT with Adaptive Width and Depth

Architectures AIMech:DynaBERTDynamicBERTWithAdaptiveWidthAndDepth DRAFT

Aliases: DynaBERT, DynaBERT Adaptive Width and Depth Transformer, Dynamic Width-and-Depth BERT with Network Rewiring

A dynamic Transformer architecture and two-stage distillation training method that adapts a task-specific BERT or RoBERTa model across both width multipliers m_w (retaining the leftmost floor(m_w * N_H) attention heads and floor(m_w * d_ff) intermediate FFN neurons per layer) and depth multipliers m_d (retaining a balanced subset of floor(m_d * L) Transformer layers) within a single weight-sharing network by (1) rewiring attention heads and FFN neurons in descending order of first-order Taylor loss-variation importance, (2) distilling the rewired teacher into a width-adaptive teacher assistant (DynaBERT_W), and (3) distilling DynaBERT_W into sub-networks spanning all (m_w, m_d) combinations over embeddings, matched hidden states, and prediction logits.

Identity Notes & Concept Scope Introduced by Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu (NeurIPS 2020, arXiv:2004.04037). Distinct from fixed-size BERT compression methods such as TinyBERT (Round 800), MiniLM (Round 801), BERT-PKD (Round 802), DistilBERT (Round 803), MobileBERT (Round 804), and BERT-of-Theseus (Round 805), and from depth-only early-exit or layer-dropping methods such as FastBERT (Round 807), PABEE (Round 808), and LayerDrop: DynaBERT adapts both Transformer width (via parallel head and FFN-neuron slicing in Eqs. 1-2 after Taylor-importance network rewiring) and depth (via strided layer dropping and last-layer-preserving hidden-state matching in Eq. 4) inside a single deployed checkpoint.

Claims & Grounded Evidence

4 Claims

Mechanism #r806-claim-parallel-head-ffn-slicing-and-taylor-rewiring

DynaBERT decomposes each Transformer layer into N_H parallel attention heads MHAttn(X) = sum_{h=1}^{N_H} Attn^h(X) (Eq. 1) and d_ff parallel intermediate FFN neurons FFN(A) = sum_{i=1}^{d_ff} GeLU(A*W^1_{:,i} + b^1_i)*W^2_{i,:} + b^2 (Eq. 2), ranks heads and neurons by first-order Taylor importance I_h approx |(dL/dh)*h| and I_w approx |sum_{i=1}^{2d} (dL/dw_i)*w_i| on the development set, and rewires each layer in descending importance so any width multiplier m_w retains the top floor(m_w * N_H) heads and floor(m_w * d_ff) neurons without altering the residual embedding dimension d.

supports Source: DynaBERT: Dynamic BERT with Adaptive Width and Depth · Locator: Section 2.1 (Training DynaBERT_W with Adaptive Width) & Section 2.1.1 (Network Rewiring), Equations 1-2 & Figure 2, Pages 3-4
supports Source: DynaBERT: Dynamic BERT with Adaptive Width and Depth · Locator: taylor_head_importance and prefix importance dominance verification in main()

Mechanism #r806-claim-two-stage-teacher-assistant-distillation-and-layer-matching

DynaBERT first trains a width-adaptive teacher assistant DynaBERT_W at full depth (m_d = 1) using L = lambda_1 * l_pred(y^{(m_w)}, y) + lambda_2 * (l_emb(E^{(m_w)}, E) + l_hidn(H^{(m_w)}, H)) (Eq. 3), and then distills DynaBERT_W into joint width-and-depth sub-networks (m_w, m_d) via Eq. 4, where for m_d < 1 student layers at depth d satisfying mod(d, 1/(1 - m_d)) == 0 are dropped and the kept student layers L_S are matched via MSE against teacher layers L_T satisfying mod(d + 1, 1/(1 - m_d)) != 0 (preserving the final teacher layer L).

supports Source: DynaBERT: Dynamic BERT with Adaptive Width and Depth · Locator: Section 2.1.2 (Eq. 3), Section 2.2 (Eq. 4), and Appendix A (Layer Pruning Strategy and Hidden States Matching), Pages 4-5 & 13
supports Source: DynaBERT: Dynamic BERT with Adaptive Width and Depth · Locator: student_dropped_layers and teacher_matched_layers verification for m_d=0.75 and m_d=0.5 in main()

Application #r806-claim-glue-and-squad-multi-configuration-efficiency

Across 4 x 3 = 12 sub-network configurations (m_w in {1.0, 0.75, 0.5, 0.25}, m_d in {1.0, 0.75, 0.5}) on GLUE and SQuAD v1.1, DynaBERT and DynaRoBERTa match or exceed BERT_BASE and RoBERTa_BASE at full size (e.g., 89.7 vs. 88.7 SQuAD v1.1 dev F1 and 88.8 F1 at 0.5x depth) while consistently outperforming DistilBERT, TinyBERT, and LayerDrop under equal parameter, FLOPs, GPU latency, and Kirin 810 ARM CPU latency budgets.

supports Source: DynaBERT: Dynamic BERT with Adaptive Width and Depth · Locator: Section 3.1-3.3, Tables 1-5 & Figures 3-4, Pages 5-8

Limitation #r806-claim-fixed-residual-embedding-dimension-limitation

Because residual skip connections link the embedding dimension d across all Transformer layers, standard task-specific DynaBERT adapts only the number of attention heads N_H and intermediate FFN neurons d_ff while keeping the embedding and residual hidden state dimension d fixed.

supports Source: DynaBERT: Dynamic BERT with Adaptive Width and Depth · Locator: Section 2.1 (paragraph following Equation 2), Page 3

⚠️ Curated Limitations & Negative Results

Residual skip connections across Transformer layers prevent scaling the embedding/residual hidden state dimension d in the standard task-specific DynaBERT formulation.
Requires a two-stage distillation and data-augmentation training loop iterating over all width multipliers (Stage 1) and all 12 width-depth pairs (Stage 2) for each downstream task.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:Distillation` Distillation	narrow_match	DynaBERT uses a two-stage knowledge distillation pipeline (via a width-adaptive teacher assistant DynaBERT_W) over embeddings, hidden states, and output logits to train weight-sharing sub-networks across multiple width and depth multipliers.
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	narrow_match	DynaBERT is a dynamic Transformer neural network architecture whose Multi-Head Attention heads and Feed-Forward Network intermediate neurons are rewired by gradient-based Taylor importance and sliced at runtime.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r806-primary-pdf`	DynaBERT: Dynamic BERT with Adaptive Width and Depth Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu	primary paper full_text	ID: https://arxiv.org/bibtex/2004.04037 sha256:92673b794cdb…
`r806-arxiv-abs`	DynaBERT: Dynamic BERT with Adaptive Width and Depth (arXiv abstract) Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu	primary paper abstract_only	ID: https://arxiv.org/abs/2004.04037 sha256:2acbc2d4a0fe…
`r806-metadata-cr`	DynaBERT: Dynamic BERT with Adaptive Width and Depth (Crossref metadata) Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu	primary paper abstract_only	ID: https://arxiv.org/bibtex/2004.04037 sha256:c175b3102913…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T14:49:00Z

Updated: 2026-10-19T14:49:00Z

Source File: data/curated/DynaBERTDynamicBERTWithAdaptiveWidthAndDepth.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories