HyperPrompt (HyperNetwork-Generated Task-Conditioned Self-Attention Prompts)

At each Transformer block m for task tau, HyperPrompt-Global fuses a task embedding k_tau in R^{t'} and layer embedding z_m in R^{t'} via a two-layer ReLU MLP I^m_tau = h_t(k_tau, z_m) in R^t, uses shared global HyperNetworks H_k and H_v to generate local bottleneck matrices (U^m_{tau,k}, D^m_{tau,k}) = (W^{U_k}, W^{D_k}) I^m_tau and (U^m_{tau,v}, D^m_{tau,v}) = (W^{U_v}, W^{D_v}) I^m_tau (Eqs. 7-8), projects the task global prompt P_tau in R^{l x d} into P^m_{tau,k} = U^m_{tau,k}(ReLU(D^m_{tau,k}(P_tau))) and P^m_{tau,v} = U^m_{tau,v}(ReLU(D^m_{tau,v}(P_tau))) (Eqs. 5-6), and prepends them to the original keys and values as K'_tau = concat(P^m_{tau,k}, K_tau) and V'_tau = concat(P^m_{tau,v}, V_tau) (Eqs. 1-2) so O_tau = softmax(Q_tau K'_tau^T) V'_tau in R^{L x d} retains sequence length L and skips passing prompt tokens through standard FFN layers.

HyperPrompt-Global introduces exactly d l T + 4(b d t) + T t' + M t' + (2 t' + t) e additional task-conditioning parameters (where d is model dimension, l is prompt length, T is number of tasks, b is local bottleneck dimension, M is number of layers, t'/t are raw/final task embedding dimensions, and e is MLP hidden dimension), yielding O(d(l T + 4 b t)) space complexity that is independent of layer count M in its dominant terms and scales sub-linearly with T while requiring 9.8 x 10^{12} forward-pass ops on T5-Base versus 3.14 x 10^{13} for HyperFormer++.

Across T5 model sizes from Base (220M) to XXL (11B), full multi-task fine-tuning with HyperPrompt-Global achieves state-of-the-art multi-task accuracy on GLUE and SuperGLUE, reaching 86.8 GLUE / 78.9 SuperGLUE on T5-Base (vs. 85.5 / 77.2 for MTL and 86.5 / 78.2 for HyperFormer++), 89.4 GLUE / 87.0 SuperGLUE on T5-Large, and 91.3 SuperGLUE on T5-XXL (vs. 90.2 for MTL) with as little as 0.14% additional task-conditioning parameters.

When the Transformer backbone parameters theta are kept frozen and only task-conditioned parameters are tuned, HyperPrompt-Global suffers a 5.5-point average score drop on SuperGLUE with T5-Large (falling from 87.0 to 81.5 in Table 1), and inserting hyper-prompts into the encoder alone degrades SuperGLUE performance on T5-Base relative to decoder-only insertion (76.5 vs. 78.9 in Table 5).