MAD-X Multiple Adapters for Cross-Lingual Transfer
Architectures AIMech:MADXMultipleAdaptersForCrossLingualTransfer DRAFT
Aliases: MAD-X, Multiple Adapters for Cross-Lingual Transfer, MAD-X Invertible Adapter Framework
A modular adapter-based architecture and transfer framework for pre-trained multilingual Transformers that combines three distinct adapter modules while keeping backbone parameters frozen: (1) language adapters LA_l(h_l, r_l) = U_l(ReLU(D_l(h_l))) + r_l trained via masked language modeling (MLM) on unlabeled monolingual corpora, (2) task adapters TA_l(h_l, r_l) = U_l(ReLU(D_l(LA_l))) + r_l stacked atop the fixed source-language adapter during downstream task training, and (3) NICE-coupled invertible adapters A_inv and A_inv^{-1} wrapping tied input and output token embeddings, enabling zero-shot cross-lingual transfer by substituting target-language invertible and language adapters at inference.
Identity Notes & Concept Scope
Introduced by Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder (EMNLP 2020, pp. 7654-7673, arXiv:2005.00052) as MAD-X (Multiple ADapters for Cross-lingual transfer). Distinct from AdapterFusion (AIMech:AdapterFusion), which combines multiple task adapters via a softmax attention layer rather than stacking language, task, and invertible adapters for cross-lingual transfer; and from LT-SFT (AIMech:LTSFTLotteryTicketSparseFineTuning), which composes sparse parameter difference vectors in weight space without adding adapter layers.
Claims & Grounded Evidence
4 Claims
Mechanism
#madx-stacked-language-and-task-bottleneck-adapters
At each Transformer layer l, MAD-X inserts a language adapter LA_l(h_l, r_l) = U_l(ReLU(D_l(h_l))) + r_l after the feed-forward layer normalization (with FFN residual r_l) and stacks a task adapter TA_l(h_l, r_l) = U_l^TA(ReLU(D_l^TA(LA_l))) + r_l on top, training LA_l via MLM on unlabeled language data and TA_l on labeled source-language task data while keeping LA_l fixed, then swapping LA_l^src with LA_l^trg at inference for zero-shot transfer.
Mechanism
#madx-nice-coupled-invertible-embedding-adapters
To adapt tied input and output token embeddings without overfitting output embeddings to MLM, MAD-X splits each h-dimensional token embedding e = [e_1, e_2] into e_1, e_2 in R^{h/2} and applies an exactly invertible NICE coupling layer o_1 = F(e_2) + e_1, o_2 = G(o_1) + e_2 after the input embeddings and its exact inverse e_2 = o_2 - G(o_1), e_1 = o_1 - F(e_2) before the tied output embeddings, where F and G are ReLU bottleneck projections with hidden dimension h/4.
Application
#madx-cross-lingual-ner-xcopa-xquad-transfer
Adding only 3.05% parameters per language (8.25M parameters on XLM-R Base), MAD-X improves average cross-lingual WikiANN NER F1 across all 16x16 source-target language pairs (including 8 unseen low-resource languages) from 32.6 to 38.2 (+5.6 F1) and zero-shot transfer from English from 41.6 to 53.2 (+11.6 F1), while outperforming XLM-R on XCOPA causal commonsense reasoning (61.5 vs. 59.7).
Limitation
#madx-task-only-unseen-language-and-monolingual-corpus-limitation
Using task adapters alone without language and invertible adapters (MAD-X - LAD - INV) degrades significantly on unseen languages (29.8 vs. 38.2 average NER F1), and MAD-X requires unlabeled monolingual corpora in each target language to pretrain its language and invertible adapters via MLM.
⚠️ Curated Limitations & Negative Results
- Requires unlabeled monolingual text in each target language to train language and invertible adapters via masked language modeling before zero-shot task transfer.
- Without language and invertible adapters, task adapters alone lack the capacity to bridge script and vocabulary mismatches on languages unseen during multilingual pretraining.
- Inserted serial language, task, and invertible adapter layers add sequential operations to every Transformer layer during inference.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | MAD-X is a modular neural adapter architecture for cross-lingual transfer in pre-trained multilingual Transformer networks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r713-primary-pdf
|
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=MAD-X+An+Adapter-Based+Framework+for+Multi-Task+Cross-Lingual+Transfer&h=5&format=xml
sha256:d7cc5f7342ff…
|
r713-arxiv-abs
|
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2005.00052
sha256:2466a94efab2…
|
r713-metadata-cr
|
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=MAD-X+An+Adapter-Based+Framework+for+Multi-Task+Cross-Lingual+Transfer&h=5&format=xml
sha256:aba0280ee8db…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T13:16:00Z
Updated: 2026-10-19T13:16:00Z
Source File:
data/curated/MADXMultipleAdaptersForCrossLingualTransfer.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories