Home / Curated / Algorithms / BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Algorithms AIMech:BERToftheseusCompressingBERTByProgressiveModuleReplacing DRAFT

Aliases: BERT-of-Theseus, Theseus Compression for BERT, Progressive Module Replacing for Transformer Compression

A distillation-loss-free Transformer compression algorithm ('Theseus Compression') that partitions a frozen N-layer predecessor network P = {prd_1, ..., prd_n} and a trainable M-layer successor network S = {scc_1, ..., scc_n} (with M < N) into n matched modules, stochastically replaces each predecessor module with its corresponding successor module during training forward passes via independent Bernoulli variables r_{i+1} ~ Bernoulli(p_d) so that y_{i+1} = r_{i+1} * scc_i(y_i) + (1 - r_{i+1}) * prd_i(y_i), increases the replacement probability according to a linear curriculum schedule p_d(t) = min(1, k*t + b) while optimizing only the standard task-specific objective, and subsequently fine-tunes the standalone successor S at p_d = 1.

Identity Notes & Concept Scope Introduced by Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou (EMNLP 2020, pp. 7859-7869, arXiv:2002.02925, doi:10.18653/v1/2020.emnlp-main.633). Distinct from knowledge distillation methods such as TinyBERT (Round 800), MiniLM (Round 801), BERT-PKD (Round 802), DistilBERT (Round 803), and MobileBERT (Round 804): BERT-of-Theseus requires zero auxiliary feature-matching, attention-matching, or logit KL-divergence losses and no balancing hyperparameters between task and distillation losses, relying instead on closed-loop gradient interactions between frozen predecessor modules and trainable successor modules inside the hybrid computational graph.

Claims & Grounded Evidence

4 Claims

Mechanism #r805-claim-stochastic-module-replacement-dynamics

Given a frozen predecessor P partitioned into n multi-layer modules {prd_1, ..., prd_n} and a successor S of n single-layer modules {scc_1, ..., scc_n}, BERT-of-Theseus samples independent binary indicators r_{i+1} ~ Bernoulli(p_d) at each module boundary i and computes y_{i+1} = r_{i+1} * scc_i(y_i) + (1 - r_{i+1}) * prd_i(y_i) (Eqs. 2-3), updating only successor parameters via backpropagation of the pure task cross-entropy loss L = CrossEntropy(y_hat, y) (Eq. 4) before a second successor-only fine-tuning phase (Eq. 5).

supports Source: BERT-of-Theseus: Compressing BERT by Progressive Module Replacing · Locator: Section 2.1 (Workflow of Theseus Compression), Equations 1-5 & Figure 1, Pages 2-3
supports Source: BERT-of-Theseus: Compressing BERT by Progressive Module Replacing · Locator: module_step and enumerate_hybrid_expectation over 2^6=64 hybrid paths in main()

Mechanism #r805-claim-curriculum-schedule-and-lr-warmup-equivalence

To bridge the transition from hybrid module replacement to standalone successor inference, BERT-of-Theseus applies a linear curriculum replacement scheduler p_d(t) = min(1, k*t + b) (Eq. 6) with slope k > 0 and base probability b in (0, 1); in expectation over Bernoulli sampling, this curriculum scales the effective learning rate of each successor module as lr'(t) = min(1, k*t + b) * lr (Eq. 7), acting as an implicit module-level learning-rate warmup.

supports Source: BERT-of-Theseus: Compressing BERT by Progressive Module Replacing · Locator: Section 2.2 (Curriculum Replacement) & Section 2.3 (A Unified View of Dropout, Warmup and Curriculum Replacement), Equations 6-7, Pages 3-4
supports Source: BERT-of-Theseus: Compressing BERT by Progressive Module Replacing · Locator: replacement_prob and effective learning rate warmup verification in main()

Application #r805-claim-glue-compression-performance

Compressing a 12-layer BERT-base predecessor into a 6-layer successor (1.94x inference speedup, 66M parameters), BERT-of-Theseus achieves a 78.6 macro-average score on the GLUE test benchmark, retaining 98.3% of the BERT-base teacher's performance and outperforming task-specific Vanilla KD (76.4), BERT-PKD (77.0), and pre-training DistilBERT (76.2 on the matched 8-task GLUE set).

supports Source: BERT-of-Theseus: Compressing BERT by Progressive Module Replacing · Locator: Section 3.3 (Experimental Results), Table 1 & Table 2, Pages 4-5

Limitation #r805-claim-inter-module-dimension-matching-limitation

Because each successor module scc_i must be a drop-in functional replacement for the corresponding predecessor module prd_i within the same forward pass, the input and output hidden dimensions of scc_i and prd_i must match at every module boundary, preventing arbitrary hidden-width reduction without additional adapter projections.

supports Source: BERT-of-Theseus: Compressing BERT by Progressive Module Replacing · Locator: Section 2.1 (Workflow of Theseus Compression, Equations 2-3), Page 2

⚠️ Curated Limitations & Negative Results

Each successor module scc_i must share identical input and output tensor dimensions with its replaced predecessor module prd_i at every module boundary.
Task-specific compression requires executing the two-stage progressive replacement and successor fine-tuning procedure for each target downstream task.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Compresses deep Transformer neural networks by progressive stochastic substitution of multi-layer predecessor modules with compact successor modules.
`AIO:Distillation` Distillation	related_match	Transfers module-level functional behavior from a large frozen predecessor network to a compact successor network through curriculum hybrid forward passes without auxiliary distillation losses.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r805-primary-pdf`	BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou	primary paper full_text	ID: https://arxiv.org/bibtex/2002.02925 sha256:cece57da8bb1…
`r805-arxiv-abs`	BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (arXiv abstract) Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou	primary paper abstract_only	ID: https://arxiv.org/abs/2002.02925 sha256:254c8115962a…
`r805-metadata-cr`	BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (Crossref metadata) Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou	primary paper abstract_only	ID: https://arxiv.org/bibtex/2002.02925 sha256:a9074bf8594d…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T14:48:00Z

Updated: 2026-10-19T14:48:00Z

Source File: data/curated/BERToftheseusCompressingBERTByProgressiveModuleReplacing.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories