ATTEMPT (Attentional Mixtures of Prompt Tuning)

Given an input embedding sequence X in R^{l x d}, t frozen pre-trained source prompts P_1, ..., P_t in R^{m x d}, and a newly initialized target prompt P_target = P_{t+1} in R^{m x d}, ATTEMPT applies dimension-wise max-pooling to obtain X_hat in R^d and P_hat_j in R^d, projects X_hat through a bottleneck attention network G with H_down = W_down^T(X_hat), H_up = W_up^T(SiLU(H_down)), and H_out = LayerNorm(H_up) (W_down in R^{d x r}, W_up in R^{r x d}, r < d), computes temperature-scaled softmax weights a_j = exp(P_hat_j H_out / T) / sum_{k=1}^{t+1} exp(P_hat_k H_out / T) (Eq. 3, with T = d exp(1)), and forms the instance-wise soft prompt P_instance(X) = P_target + sum_{j=1}^{t+1} a_j P_j (Eq. 4), assigning effective weight 1 + a_{t+1} in (1, 2) to P_target so the target prompt always receives non-vanishing gradient updates even when no source task is relevant.

When sharing the attention module G (with 2 r d + 2 d parameters) across N target tasks, ATTEMPT introduces exactly d m + (2 r d + 2 d) / N = d(m + 2(r + 1) / N) trainable parameters per target task, which is strictly independent of the number of Transformer layers in the backbone language model (updating 96k parameters per task on GLUE with T5-base and 172k parameters per task with T5-XL, compared to 6M for Adapter and 2M for BitFit on T5-XL).

Evaluated across 21 NLP datasets using 6 high-resource source tasks (MNLI, QNLI, QQP, SST-2, SQuAD, ReCoRD) with a frozen T5-base backbone, ATTEMPT-m achieves 85.8% average score on GLUE (outperforming Prompt Tuning at 72.2%, SPoT at 82.3%, HyperFormer at 84.8%, HyperDecoder at 83.7%, and full fine-tuning at 84.9% while updating 96k vs. 220M parameters per task), 74.1% average on SuperGLUE, 72.8 average F1 on MRQA 2019, and outperforms fine-tuning, Adapter, SPoT, and HyperFormer in 4/16/32-shot domain adaptation on BoolQ and CB.

Because ATTEMPT prepends m = 100 soft prompt tokens to the input sequence and loads multiple source prompts alongside the attention module, it increases training peak GPU memory footprint on T5-base (13.7 GB vs. 8.5 GB for Prompt Tuning and 5.9 GB for Adapter at batch size 32) and can experience negative transfer if unrelated QA source prompts (SQuAD, ReCoRD) are included on certain target tasks such as RTE.