Home / Curated / Algorithms / WARP (Word-level Adversarial ReProgramming)

WARP (Word-level Adversarial ReProgramming)

Algorithms AIMech:WARPWordLevelAdversarialReProgramming DRAFT

Aliases: WARP, WARP_init, Word-level Adversarial ReProgramming, Word-Level Adversarial Reprogramming for Masked Language Models

A parameter-efficient transfer learning method for frozen masked language models that inserts K special prompt tokens [P_1], ..., [P_K] and a [MASK] token before, between, and after input text segments according to a template T_{Theta^P}(x), removes the full vocabulary decoder projection from the masked language modeling head f(.), and optimizes only the continuous prompt embeddings Theta^P in R^{K x E} and class verbalizer embeddings Theta^V in R^{|C| x E} in word-embedding space R^E via stochastic gradient descent on the cross-entropy loss of P_Theta(y | x) = exp(Theta^V_y f(T_{Theta^P}(x))) / sum_{i in C} exp(Theta^V_i f(T_{Theta^P}(x))).

Identity Notes & Concept Scope Introduced by Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May (ACL/IJCNLP 2021, Volume 1: Long Papers, pp. 4921-4933, arXiv:2101.00121). Inspired by image-level Adversarial Reprogramming (Elsayed et al., ICLR 2019). Distinct from discrete prompt search such as AutoPrompt (Shin et al., EMNLP 2020), which restricts triggers and verbalizers to discrete vocabulary tokens, and distinct from generative Prompt Tuning (Lester et al., EMNLP 2021; AIMech:PromptTuning), which prepends prefix prompts to sequence-to-sequence models without jointly optimizing continuous MLM verbalizer vectors Theta^V at a template [MASK] position.

Claims & Grounded Evidence

4 Claims

Mechanism #warp-continuous-prompt-and-verbalizer-mlm-reprogramming-mechanism

Given a frozen masked language model with word embedding dimension E and pre-decoder MLM head representation f(.) in R^E at the inserted [MASK] position, WARP leaves all Transformer layers and original input token embeddings untouched while optimizing only K continuous prompt vectors Theta^P = ([P_1], ..., [P_K]) in R^{K x E} and C = |C| continuous class verbalizer vectors Theta^V = ([V_1], ..., [V_C]) in R^{C x E} to maximize log P_Theta(y | x) = log(exp(Theta^V_y f(T_{Theta^P}(x))) / sum_{i in C} exp(Theta^V_i f(T_{Theta^P}(x)))).

supports Source: WARP: Word-level Adversarial ReProgramming · Locator: Section 3 WARP, Figure 3 & equations, pp. 3-4; Section 3.1 Method, p. 4

Scope #warp-multi-task-serving-storage-and-batched-inference-scope

To serve N classification tasks with at most C classes each using a frozen language model of M parameters and word embedding dimension E, WARP stores exactly M + N E (C + K) parameters (Table 4; requiring <25K trainable parameters per task for RoBERTa-large with E = 1024, K = 20, and only 1,024 trainable parameters per model for ALBERT-xxlarge-v2 with E = 128, K + C = 8) and enables mixed-task minibatch inference in a single shared forward pass by concatenating task verbalizer matrices into a single linear layer with at most N C output neurons.

supports Source: WARP: Word-level Adversarial ReProgramming · Locator: Section 4.2 Table 2, p. 6; Section 5.1 & 5.4, pp. 6-7; Section 6.3 Real-world applications, Table 4, pp. 8-9

Application #warp-glue-leaderboard-and-fewglue-32shot-application

On the GLUE evaluation server with frozen RoBERTa-large, WARP (<25K trainable parameters per task) achieves an 81.6 test score, outperforming BERT Adapters (80.2 with 1.2M trainable parameters), BERT_large (80.5 with 355M), TinyBERT_6 (78.1 with 67M), and ELECTRA_small (77.4 with 14M), while on SST-2 dev a single prompt token (WARP_1) reaches 93.8% accuracy (vs. 91.2% for AutoPrompt), and in 32-shot SuperGLUE with ALBERT-xxlarge-v2, manually initialized WARP_init achieves 70.2 F1 / 82.4% accuracy on CB and 69.1% on RTE, outperforming 175B-parameter GPT-3 (52.0 / 75.6% on CB and 69.0% on RTE).

supports Source: WARP: Word-level Adversarial ReProgramming · Locator: Section 4.2 Results, Tables 1-2, pp. 5-6; Section 5.4 Results, Table 3, p. 7; Section 6.2 Comparison with AutoPrompt, Figure 4, p. 8

Limitation #warp-cola-reformulation-and-unlabeled-distillation-gap-limitation

WARP exhibits weaker performance on linguistic acceptability tasks that are difficult to reformulate as Cloze completions (53.9 Matthews correlation on CoLA test vs. 67.8 for fully fine-tuned RoBERTa in Table 1, with learned prompt vectors remaining near their [MASK] initialization), and in 32-shot SuperGLUE trails semi-supervised iPET (82.4% vs. 88.8% on CB test) which fine-tunes all 255M parameters and distills over 20,000 unlabeled examples.

supports Source: WARP: Word-level Adversarial ReProgramming · Locator: Section 4.2 Results, Table 1, p. 5; Section 5.4 Results, Table 3, p. 7; Section 6.1 Interpreting tokens learned by WARP, p. 7; Appendix B Table 6, pp. 12-13

⚠️ Curated Limitations & Negative Results

Underperforms full fine-tuning on grammatical acceptability tasks such as CoLA (53.9 vs. 67.8 test Matthews correlation) that do not naturally map to Cloze-style masked token prediction.
Optimized continuous prompt embeddings rarely correspond to interpretable natural-language words in vocabulary space.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	WARP reprograms a frozen pretrained masked language model neural network by optimizing continuous input-level prompt embeddings and output verbalizer vectors.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r709-primary-pdf`	WARP: Word-level Adversarial ReProgramming Karen Hambardzumyan, Hrant Khachatrian, Jonathan May	primary paper full_text	ID: https://dblp.org/search/publ/api?q=WARP+Word-level+Adversarial+ReProgramming&format=xml sha256:3834b52fcfb1…
`r709-arxiv-abs`	WARP: Word-level Adversarial ReProgramming (arXiv abstract) Karen Hambardzumyan, Hrant Khachatrian, Jonathan May	primary paper abstract_only	ID: https://arxiv.org/abs/2101.00121 sha256:b668e72b0bc6…
`r709-metadata-cr`	WARP: Word-level Adversarial ReProgramming (Crossref metadata) Karen Hambardzumyan, Hrant Khachatrian, Jonathan May	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=WARP+Word-level+Adversarial+ReProgramming&format=xml sha256:3d36f760a49c…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:12:00Z

Updated: 2026-10-19T13:12:00Z

Source File: data/curated/WARPWordLevelAdversarialReProgramming.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories