Instance-Dependent Prompt Generation
Algorithms AIMech:IDPGInstanceDependentPromptGeneration DRAFT
Aliases: IDPG, M-IDPG-PHM, S-IDPG-PHM, Multi-Layer Instance-Dependent Prompt Generation
A conditional parameter-efficient prompt-tuning method for pre-trained language models that generates continuous soft prompts W_p(T, x_i) = G(M(x_i), T) in R^{t x d} conditioned on each input sequence representation M(x_i) in R^d using a lightweight two-layer bottleneck perceptron G whose down-projection W_1 in R^{m x d} and up-projection W_2 in R^{(t d) x m} are decomposed via Parameterized Hypercomplex Multiplication (PHM) Kronecker sums W = sum_{j=1}^n A_j otimes B_j, with multi-layer variant M-IDPG sharing W across all Transformer layers while maintaining layer-specific bias prompts.
Identity Notes & Concept Scope
Introduced by Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, and Hao Ma (NAACL 2022, arXiv:2204.04497) as Instance-Dependent Prompt Generation (IDPG / S-IDPG-PHM / M-IDPG-PHM). Distinct from static Prompt Tuning and SPoT (AIMech:SPoTSoftPromptTransfer), which use a fixed input-independent prompt per task (recovered as the degenerate case W = 0 in IDPG where only the second-layer bias is active); from ATTEMPT (AIMech:ATTEMPTAttentionalMixturesOfSoftPrompts), which interpolates pre-trained source-task prompts via sub-query attention rather than generating prompts through a PHM bottleneck MLP; and from Hyperdecoder (AIMech:HyperdecoderInstanceSpecificDecoderAdapter), which generates decoder adapter weights from encoder hidden states rather than layer-wise soft prompt tokens.
Claims & Grounded Evidence
4 Claims
Mechanism
#idpg-bottleneck-prompt-generator-and-static-degeneracy
Given a sentence representation M(x_i) in R^d of input x_i, IDPG generates a t-token continuous prompt W_p(T, x_i) = G(M(x_i), T) via a two-layer bottleneck generator with hidden size m << d requiring m(d+1) + t d(m+1) parameters before compression, and strictly generalizes task-specific prompt tuning since setting the projection weights W to zero leaves the second-layer bias term t x d as a static task-specific prompt.
Mechanism
#idpg-phm-kronecker-decomposition-and-multilayer-sharing
Replacing the dense generator matrices W_1 in R^{m x d} and W_2 in R^{(t d) x m} with Parameterized Hypercomplex Multiplication (PHM) layers W = sum_{i=1}^n A_i otimes B_i (with A_i in R^{n x n} shared across layers) reduces single-layer generator parameters from 1.5M (S-IDPG-DNN) to 105K (S-IDPG-PHM) for d=1024, m=256, t=5, n=16, while multi-layer M-IDPG-PHM shares W across N=24 layers with layer-specific biases in 134K trainable parameters (0.04% of RoBERTa-Large).
Application
#idpg-nlu-and-few-shot-benchmark-performance
Across ten NLU benchmarks (MPQA, Subj, CR, MR, SST-2, QNLI, RTE, MRPC, STS-B, QQP) on RoBERTa-Large, M-IDPG-PHM achieves 91.9 average score with 134K trainable parameters, outperforming standard prompt tuning (88.8) by +3.1 points and P-tuning v2 (90.3) by +1.6 points, and in few-shot settings (K in {100, 500, 1000}) outperforms prompt tuning by +2.5, +2.1, and +2.0 points respectively.
Limitation
#idpg-two-forward-pass-and-low-resource-variance-limitation
When using the pre-trained language model itself to compute M(x_i), IDPG requires two forward passes at inference time unless M(x_i) is cached or replaced by lightweight GloVe embeddings (M-IDPG-PHM-GloVe), and exhibits elevated initialization variance in extreme low-resource regimes (e.g., standard deviation 4.6 on MPQA at K=100).
⚠️ Curated Limitations & Negative Results
- Computing the instance representation M(x_i) with the pre-trained Transformer requires an extra forward pass at inference time unless M(x_i) is pre-cached or substituted with static word embeddings such as GloVe.
- In extreme few-shot regimes (K = 100), generator initialization sensitivity can cause higher run-to-run variance on small datasets such as MPQA.
- Combining residual connections and LayerNorm simultaneously inside the single-layer PHM prompt generator degrades accuracy compared to PHM alone or PHM with LayerNorm only.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | IDPG is a conditional prompt-generation architecture and fine-tuning algorithm for frozen pre-trained Transformer neural networks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r711-primary-pdf
|
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, Hao Ma
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=IDPG+An+Instance-Dependent+Prompt+Generation+Method&h=5&format=xml
sha256:bc7419642a39…
|
r711-arxiv-abs
|
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, Hao Ma
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2204.04497
sha256:0fe366159904…
|
r711-metadata-cr
|
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, Hao Ma
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=IDPG+An+Instance-Dependent+Prompt+Generation+Method&h=5&format=xml
sha256:86944201b509…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T13:14:00Z
Updated: 2026-10-19T13:14:00Z
Source File:
data/curated/IDPGInstanceDependentPromptGeneration.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories