LayerDrop: Structured Dropout for On-Demand Transformer Depth Reduction
Algorithms AIMech:LayerDropStructuredDropoutForTransformerDepthReduction DRAFT
Aliases: LayerDrop, LayerDrop Structured Layer Dropout for Transformers, On-Demand Transformer Depth Pruning via LayerDrop
A structured dropout regularization and on-demand depth-pruning algorithm for Transformer networks that constrains a binary Bernoulli mask M in {0, 1} to be constant across each predefined layer weight group G in calG (so M[i] = M[j] for all (i, j) in G and entire layers are stochastically bypassed via residual connections with drop probability p during training, yielding N(1 - p) expected active layers), sets the optimal training drop rate to p* = 1 - r/N for a target inference depth of r layers, and extracts shallow sub-networks of any desired depth at inference time without post-hoc fine-tuning by deterministically dropping every other strided layer d == 0 (mod floor(1/p_prune)).
Identity Notes & Concept Scope
Introduced by Angela Fan, Edouard Grave, and Armand Joulin (ICLR 2020, arXiv:1909.11556). Distinct from unstructured weight DropConnect (which zeroes individual weights without reducing inference latency), from Stochastic Depth in ResNets (which uses a linear depth-decay survival schedule purely to accelerate training without inference-time layer pruning), and from distillation-based depth compression methods such as DistilBERT (Round 803) and TinyBERT (Round 800) that must train a separate student model for each target depth.
Claims & Grounded Evidence
4 Claims
Mechanism
#r810-claim-group-constant-structured-dropout-and-optimal-drop-rate
LayerDrop generalizes DropConnect W_d = M o W by partitioning network parameters into predefined layer groups G in calG and enforcing M[i] = M[j] for all (i, j) in G with independent group drop probability p (Section 3.2.1); because N(1 - p) layers are active on average during training, targeting an inference-time pruned sub-network of r layers out of N yields the optimal training drop rate p* = 1 - r/N (Section 3.2.2).
Mechanism
#r810-claim-every-other-strided-inference-pruning
At inference time, LayerDrop extracts shallow sub-networks from a single trained Transformer without fine-tuning using the deterministic 'Every Other' strategy, which prunes at rate p_prune by dropping layers at depth d satisfying d == 0 (mod floor(1/p_prune)), matching or nearly matching validation-search and learned data-driven softmax layer-selection while avoiding the severe perplexity degradation caused by contiguous first-half or last-half layer removal.
Application
#r810-claim-deep-transformer-stabilization-and-roberta-pruning
LayerDrop stabilizes and regularizes very deep Transformers—enabling a 12-encoder-layer Transformer to reach 30.2 BLEU on WMT14 En-De (where the baseline without LayerDrop diverges), a 40-layer Adaptive Inputs Transformer to reach 17.7 test perplexity on WikiText-103, and a 48-layer RoBERTa-Large model to train at almost 2x speed with p = 0.5—while 6-layer (50% pruned) and 3-layer (75% pruned) RoBERTa sub-networks outperform DistilBERT and from-scratch small models on MNLI and SST-2.
Limitation
#r810-claim-boundary-layer-sensitivity-and-high-pruning-rate-limitation
In a LayerDrop-trained Transformer, the earliest input layers and final output projection layers remain disproportionately critical to performance (removing contiguous blocks of initial or final layers causes severe perplexity degradation from 19.7 to 26.1-29.3 on WikiText-103), and extracting aggressively pruned sub-networks (e.g., >=50% depth reduction) requires training with a correspondingly large drop rate (p = 0.4-0.5).
⚠️ Curated Limitations & Negative Results
- Input-adjacent and output-adjacent Transformer layers remain critical for token processing and vocabulary projection, so pruning contiguous initial or final layer blocks severely degrades accuracy.
- Aggressive inference-time depth reduction (>=50% layers pruned) requires training with a high LayerDrop rate (p = 0.4-0.5) to align train-time and test-time active depths.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | Regularizes deep Transformer neural networks via group-constant layer dropout masks during training to enable zero-shot structured depth pruning at inference time. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r810-primary-pdf
|
Angela Fan, Edouard Grave, Armand Joulin
|
primary paper
full_text
|
ID: https://arxiv.org/bibtex/1909.11556
sha256:e18966beebdb…
|
r810-arxiv-abs
|
Angela Fan, Edouard Grave, Armand Joulin
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1909.11556
sha256:1fbc269f83e9…
|
r810-metadata-cr
|
Angela Fan, Edouard Grave, Armand Joulin
|
primary paper
abstract_only
|
ID: https://arxiv.org/bibtex/1909.11556
sha256:2b9ddb7be4da…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T14:53:00Z
Updated: 2026-10-19T14:53:00Z
Source File:
data/curated/LayerDropStructuredDropoutForTransformerDepthReduction.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories