Home / Curated / Algorithms / PPT Pre-Trained Prompt Tuning for Few-Shot Learning

PPT Pre-Trained Prompt Tuning for Few-Shot Learning

Algorithms AIMech:PPTPretrainedPromptTuningForFewShotLearning DRAFT

Aliases: PPT, Pre-trained Prompt Tuning, Hybrid PPT and Unified PPT, Self-Supervised Pre-Trained Soft Prompt Tuning

A parameter-efficient few-shot prompt-tuning framework that groups downstream classification tasks into format-specific or unified multiple-choice Pattern-Verbalizer Pairs PVP_i^pre = (f_i^pre, v_i^pre), pre-trains a sequence of continuous soft prompt embeddings P_i on large-scale unlabeled corpora via self-supervised cloze objectives arg max_P sum_x log p(<X> = v(y) | [P; f(x)]; P) with all pretrained language model weights frozen, and uses P_i to initialize downstream few-shot soft prompt optimization.

Identity Notes & Concept Scope Distinct from vanilla Prompt Tuning (Lester et al., EMNLP 2021, AIMech:PromptTuning), which initializes soft prompts randomly or from vocabulary word embeddings and suffers high variance and poor accuracy in few-shot settings, and from SPoT (Vu et al., ACL 2022, AIMech:SPoTSoftPromptTransfer), which transfers soft prompts from supervised source tasks rather than self-supervised pre-training on unlabeled corpora via unified Pattern-Verbalizer Pairs.

Claims & Grounded Evidence

4 Claims

Mechanism #r722-claim-self-supervised-pvp-formats

PPT designs self-supervised pre-training Pattern-Verbalizer Pairs on 10GB unlabeled corpora (OpenWebText and WuDaoCorpora) for three task families—3-class document-coherent next sentence prediction for sentence-pair classification, multi-option next sentence selection for multiple-choice classification, and pseudo-labeled single-sentence classification—and further unifies all three into a single variable-option (2 to 16 options) multiple-choice pre-training format (Unified PPT).

supports Source: PPT: Pre-trained Prompt Tuning for Few-shot Learning · Locator: Section 3.2-3.3 & Appendix C.3, Equations 3-5, Table 9, pp. 4-5, 13

Mechanism #r722-claim-parameter-and-memory-efficiency

With 100 soft tokens of dimension 4096 on 11B-parameter backbones (T5-XXL and CPM-2), PPT tunes only 409,600 (~410K) parameters per task—nearly 27,000x (over 3,000x in storage) smaller than full-model tuning—while reducing per-step training time from 4,416 ms to 794 ms and GPU memory from 259 GB to 72 GB on SST-2.

supports Source: PPT: Pre-trained Prompt Tuning for Few-shot Learning · Locator: Section 4.1 & Appendix E, Table 10, pp. 6, 13-14

Application #r722-claim-few-shot-accuracy-and-variance

Across 15 English and Chinese benchmarks in the 32-sample few-shot setting on 11B-parameter T5-XXL and CPM-2 models, PPT and Unified PPT substantially outperform vanilla Prompt Tuning and LM Adaptation while dramatically reducing seed variance (e.g., improving English SST-2 from 70.5+-15.5 to 94.4+-0.3 and Chinese CCPM from 31.0+-9.7 to 85.4+-0.6, surpassing full-model fine-tuning).

supports Source: PPT: Pre-trained Prompt Tuning for Few-shot Learning · Locator: Section 4.2, Tables 4-5, pp. 6-7

Limitation #r722-claim-convergence-and-pseudo-label-limitation

Single-sentence PPT relies on an external fine-tuned classifier (e.g., RoBERTa-base trained on Yelp-5) to annotate pseudo-labels and cannot directly generalize to arbitrary non-sentiment single-text domains with many classes unless reformulated via Unified PPT, and soft prompt optimization still requires a much larger learning rate (5e-3 to 5e-2) and more training steps to converge than full-model fine-tuning.

supports Source: PPT: Pre-trained Prompt Tuning for Few-shot Learning · Locator: Section 3.2.3, Section 4.2-4.3, and Appendix C.2, Figure 5, pp. 5, 7-8, 12-13

⚠️ Curated Limitations & Negative Results

Format-specific single-sentence PPT requires an auxiliary supervised classifier to generate pseudo-labels on unlabeled text and does not generalize across domains or >5 classes without switching to Unified PPT.
Even with pre-trained initialization, soft prompt optimization converges in more steps than full-model fine-tuning and requires an upfront 200,000-step self-supervised prompt pre-training stage on 10GB of unlabeled text.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	PPT adapts frozen 11B-parameter Transformer language models to few-shot classification tasks by pre-training continuous soft prompt embeddings on unlabeled corpora.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r722-primary-pdf`	PPT: Pre-trained Prompt Tuning for Few-shot Learning Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang	primary paper full_text	ID: https://dblp.org/search/publ/api?q=PPT+Pre-trained+Prompt+Tuning+for+Few-shot+Learning+Gu&format=xml sha256:62a886a40a97…
`r722-arxiv-abs`	PPT: Pre-trained Prompt Tuning for Few-shot Learning (arXiv abstract) Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang	primary paper abstract_only	ID: https://arxiv.org/abs/2109.04332 sha256:ab4cf4abb0d4…
`r722-metadata-cr`	PPT: Pre-trained Prompt Tuning for Few-shot Learning (Crossref metadata) Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=PPT+Pre-trained+Prompt+Tuning+for+Few-shot+Learning+Gu&format=xml sha256:7532926c7316…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:25:00Z

Updated: 2026-10-19T13:25:00Z

Source File: data/curated/PPTPretrainedPromptTuningForFewShotLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories