Home / Curated / Architectures / MobileBERT: Compact Task-Agnostic BERT with Bottleneck Structures and Progressive Knowledge Transfer

MobileBERT: Compact Task-Agnostic BERT with Bottleneck Structures and Progressive Knowledge Transfer

Architectures AIMech:MobileBERTCompactTaskAgnosticBERTWithBottlenecks DRAFT

Aliases: MobileBERT, MobileBERT Bottleneck Transformer Architecture, Inverted-Bottleneck Progressive Knowledge Transfer for MobileBERT

A deep-and-thin task-agnostic 24-layer Transformer architecture and progressive distillation framework that compresses BERT width rather than depth by equipping both an Inverted-Bottleneck teacher (IB-BERT_LARGE, inter-block hidden dimension 512 and intra-block dimension 1024) and a bottlenecked student (MobileBERT, inter-block dimension 512 and intra-block dimension 128) with shared-dimension linear input/output projections, stacking 4 feed-forward networks per Multi-Head Attention block to restore the 1:2 MHA-to-FFN parameter ratio, compressing token embeddings to 128 dimensions with a kernel-3 1D convolution, replacing LayerNorm and GELU with elementwise affine NoNorm(h) = gamma o h + beta and ReLU, and transferring layer-wise feature maps and attention distributions bottom-to-top across L stages via Progressive Knowledge Transfer (PKT) followed by Pre-training Distillation.

Identity Notes & Concept Scope Introduced by Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou (ACL 2020, pp. 2158-2170, arXiv:2004.02984, doi:10.18653/v1/2020.acl-main.195). Distinct from shallow-and-wide BERT distillation methods such as DistilBERT (Round 803), BERT-PKD (Round 802), TinyBERT (Round 800), and MiniLM (Round 801): MobileBERT preserves the full 24-layer depth of BERT_LARGE to avoid the capacity loss of depth reduction and narrows intra-block hidden dimensions via matched inter-block linear bottlenecks (d_inter = 512) so that Feature Map Transfer (Eq. 2) and Attention Transfer (Eq. 3) require no auxiliary projection matrices.

Claims & Grounded Evidence

4 Claims

Mechanism #r804-claim-bottleneck-and-stacked-ffn-architecture

MobileBERT pairs a 24-layer Inverted-Bottleneck teacher (IB-BERT_LARGE with inter-block hidden size 512 and intra-block hidden size 1024) with a 24-layer bottlenecked student (inter-block hidden size 512 and intra-block hidden size 128 with 512-dimensional residual connections), and replaces the single FFN in each block with 4 stacked FFNs (128 input/output, 512 intermediate) to restore the 1:2 Multi-Head Attention-to-FFN parameter ratio distorted by the bottleneck projections.

supports Source: MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices · Locator: Section 2.1 (Bottleneck and Inverted-Bottleneck) & Section 2.2 (Stacked Feed-Forward Networks), Figure 1 & Table 1, Pages 2-3
supports Source: MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices · Locator: MHA-to-FFN parameter ratio verification (2:1 with 1 FFN vs 1:2 with 4 stacked FFNs) in main()

Mechanism #r804-claim-progressive-knowledge-transfer-equations

Because joint multi-layer transfer across 24 layers can destabilize student optimization, MobileBERT trains the student via L-stage bottom-to-top Progressive Knowledge Transfer (PKT), where stage l freezes lower layers 1..l-1 and minimizes a linear combination of Feature Map Transfer L_FMT^l = (1/(T*N)) sum_{t=1}^T sum_{n=1}^N (H_{t,l,n}^{tr} - H_{t,l,n}^{st})^2 (Eq. 2) and Attention Transfer L_AT^l = (1/(T*A)) sum_{t=1}^T sum_{a=1}^A D_KL(a_{t,l,a}^{tr} || a_{t,l,a}^{st}) (Eq. 3), followed by Pre-training Distillation L_PD = alpha * L_MLM + (1 - alpha) * L_KD + L_NSP (Eq. 4).

supports Source: MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices · Locator: Section 2.4 (Training Objectives) & Section 2.5 (Training Strategies), Equations 2-4 & Figure 2, Pages 3-4
supports Source: MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices · Locator: feature_map_transfer_loss, attention_transfer_loss, and L_PD verification in main()

Application #r804-claim-operational-optimizations-and-benchmark-performance

Replacing LayerNorm with the elementwise affine operation NoNorm(h) = gamma o h + beta (Eq. 1), GELU with ReLU, and 768-d embeddings with 128-d embeddings expanded by a kernel-3 1D convolution yields a 25.3M-parameter MobileBERT that is 4.3x smaller and 5.5x faster than BERT_BASE (62 ms vs. 342 ms latency on a Pixel 4 phone), achieving 77.7 GLUE test score (0.6 below BERT_BASE) and 90.0 / 79.2 dev F1 on SQuAD v1.1 / v2.0 (+1.5 / +2.1 F1 over BERT_BASE).

supports Source: MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices · Locator: Section 2.3 (Operational Optimizations, Eq. 1) & Section 3 (Experiments, Tables 2-4), Pages 3-6

Limitation #r804-claim-custom-ib-teacher-pretraining-cost

MobileBERT cannot directly distill intermediate layer feature maps and 4-head attention maps from an off-the-shelf standard BERT_LARGE checkpoint; it requires first pre-training a custom 24-layer Inverted-Bottleneck teacher (IB-BERT_LARGE) from scratch whose inter-block dimension (512) and head count (4) match the student, followed by 24 sequential layer-wise PKT stages.

supports Source: MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices · Locator: Section 2.1, Section 2.5 (Progressive Knowledge Transfer), and Section 3.1 (Implementation Details), Pages 2-5

⚠️ Curated Limitations & Negative Results

Requires pre-training a custom 24-layer Inverted-Bottleneck teacher (IB-BERT_LARGE) with matched inter-block dimension (512) and attention head count (4) before student distillation.
Progressive Knowledge Transfer requires L=24 sequential layer-by-layer training stages prior to Pre-training Distillation.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:Distillation` Distillation	narrow_match	MobileBERT introduces L-stage bottom-to-top Progressive Knowledge Transfer (PKT) combining Feature Map Transfer (FMT), Attention Transfer (AT), and Pre-training Distillation (PD) from an Inverted-Bottleneck teacher.
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	narrow_match	MobileBERT is a deep-and-thin 24-layer bottleneck Transformer neural network architecture with 4 stacked FFNs per Multi-Head Attention block and operational NoNorm layers.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r804-primary-pdf`	MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou	primary paper full_text	ID: https://arxiv.org/bibtex/2004.02984 sha256:5206d731107a…
`r804-arxiv-abs`	MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (arXiv abstract) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou	primary paper abstract_only	ID: https://arxiv.org/abs/2004.02984 sha256:b5bd64da56b6…
`r804-metadata-cr`	MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (Crossref metadata) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou	primary paper abstract_only	ID: https://arxiv.org/bibtex/2004.02984 sha256:1ce81e974047…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T14:47:00Z

Updated: 2026-10-19T14:47:00Z

Source File: data/curated/MobileBERTCompactTaskAgnosticBERTWithBottlenecks.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories