Home / Curated / Algorithms / MPT (Multitask Prompt Tuning)

MPT (Multitask Prompt Tuning)

Algorithms AIMech:MPTMultitaskPromptTuning DRAFT

Aliases: Multitask Prompt Tuning, MPT Prompt Decomposition and Distillation, MPT* Multitask Target Prompt Tuning

A parameter-efficient multitask prompt transfer methodology for frozen pre-trained language models that decomposes each task's soft prompt matrix into the Hadamard product of a single task-shared prompt matrix P^* and a rank-one task-specific matrix W_k = u_k (x) v_k^T, learns this decomposition during source training via logit and hidden-state knowledge distillation from independently trained teacher prompts, and adapts P^* to downstream target tasks via two-speed low-rank multiplicative updates.

Identity Notes & Concept Scope Introduced by Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim (ICLR 2023, arXiv:2303.02861). Distinct from SPoT (Vu et al., ACL 2022; AIMech Round 698), which trains a single unfactorized prompt on a source mixture or retrieves top-k source prompts by cosine similarity without multiplicative rank-one prompt decomposition or knowledge distillation from per-task teacher prompts. Both single-target adaptation (MPT, tuning (l x d) + (l + d) parameters per task) and multitask target group adaptation (MPT*, sharing P^* across tau target tasks for (l x d)/tau + (l + d) parameters per task) are covered within this record.

Claims & Grounded Evidence

4 Claims

Mechanism #mpt-hadamard-rank-one-prompt-decomposition-mechanism

MPT decomposes the soft prompt P_hat_k in R^{l x d} for task k into the Hadamard product of a slow task-shared prompt matrix P^* in R^{l x d} and a fast rank-one task-specific matrix W_k = u_k (x) v_k^T formed by outer product of vectors u_k in R^l and v_k in R^d: P_hat_k = P^* o (u_k (x) v_k^T) (Eq. 2), requiring only (l x d) + (l + d) = 77.6K tunable parameters per single target task (0.035% of T5-Base's 220M parameters, collapsible into an l x d matrix after training) and (l x d)/tau + (l + d) = 10.5K parameters per task for multitask target adaptation across tau = 8 GLUE tasks.

supports Source: Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning · Locator: Section 3.1 Prompt decomposition, Eq. 2 & Figure 3, p. 4; Section 3.2 Parameter-efficiency, p. 5; Section 4.2 Table 1, p. 6

Mechanism #mpt-logit-and-hidden-state-prompt-distillation-mechanism

To prevent the shared component P^* from overfitting to large source datasets during stochastic multitask training, MPT trains student prompts P_hat_k = P^* o (u_k (x) v_k^T) against separately trained per-task teacher prompts P_k^(teacher) by minimizing L_Total = L_PLM + lambda (L_Logits + L_Hidden) (Eq. 5), combining temperature-scaled output probability KL divergence L_Logits (Eq. 3) and encoder/decoder intermediate hidden state mean squared error L_Hidden = sum (H_{k,i} - H_{k,i}^(teacher))^2 (Eq. 4).

supports Source: Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning · Locator: Section 3.1 Prompt distillation, Eqs. 3-5, p. 4; Section 3.2 Source Training and Target Adaptation, p. 5; Section 4.3 Ablation Studies, Table 6, pp. 8-9

Application #mpt-nlu-and-nlg-benchmarks-application

Evaluated on 23 NLP datasets across GLUE (85.6 average with 77.6K params and 85.8 for MPT* with 10.5K params on T5-Base, outperforming 220M full finetuning at 84.9), SuperGLUE (74.1 single-task and 76.1 multitask, +16.3% over vanilla Prompt Tuning), MRQA, Others (WinoGrande, Yelp-2, SciTail, PAWS), 4/16/32-shot adaptation, and NLU-to-NLG transfer on E2E and WebNLG.

supports Source: Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning · Locator: Section 4.1-4.2, Tables 1-5 & Figure 4, pp. 5-8

Limitation #mpt-mrqa-gap-and-long-prompt-overfitting-limitation

MPT requires a two-step source pretraining pipeline (training separate teacher prompts for every source task before joint multitask distillation), still trails full model finetuning and 1.9M-parameter Adapters on extractive reading comprehension (MRQA: 72.2 vs. 74.7), and suffers a 1.8% SuperGLUE accuracy drop when prompt length l is increased from 300 to 400 due to overfitting.

supports Source: Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning · Locator: Section 3.2 Source Training and Target Adaptation, p. 5; Section 4.2 Results and Analysis, Table 2, pp. 6-7; Section 4.3 Prompt length, Figure 5, p. 9

⚠️ Curated Limitations & Negative Results

Source training requires first training individual teacher soft prompts on every source dataset before running joint multitask knowledge distillation.
On complex question answering benchmarks (MRQA), a noticeable accuracy gap remains between 77.6K-parameter MPT (72.2 F1) and 220M-parameter full finetuning (74.7 F1).
Increasing the prompt length l beyond 300 tokens (e.g., to l = 400) degrades downstream SuperGLUE accuracy by 1.8% due to overfitting, and freezing either P^* or (u_t, v_t) during target adaptation substantially degrades accuracy (to 71.3% or 62.5%).

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	MPT adapts frozen pre-trained Transformer language models across multiple tasks by decomposing and distilling continuous soft prompt matrices.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r702-primary-pdf`	Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, Yoon Kim	primary paper full_text	ID: https://dblp.org/search/publ/api?q=Multitask+Prompt+Tuning+Enables+Parameter-Efficient+Transfer+Learning&format=xml sha256:a7e832564edd…
`r702-arxiv-abs`	Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning (arXiv abstract) Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, Yoon Kim	primary paper abstract_only	ID: https://arxiv.org/abs/2303.02861 sha256:9198d0496877…
`r702-metadata-cr`	Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning (Crossref metadata) Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, Yoon Kim	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=Multitask+Prompt+Tuning+Enables+Parameter-Efficient+Transfer+Learning&format=xml sha256:9c040a78ddb5…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:05:00Z

Updated: 2026-10-19T13:05:00Z

Source File: data/curated/MPTMultitaskPromptTuning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories