UniPELT

A unified parameter-efficient language model tuning architecture that incorporates multiple heterogeneous parameter-efficient tuning methods—specifically bottleneck Adapters (A), Prefix-tuning (P), and Low-Rank Adaptation / LoRA (L)—as submodules operating on distinct components of each frozen Transformer layer, and dynamically controls their activation via trainable layer-wise feedforward sigmoid gating mechanisms G_{m_i} in (0, 1). In each layer, a LoRA gate G_L(h_in) in (0, 1) replaces LoRA's static scaling hyperparameter alpha to modulate low-rank query and value updates theta_task = theta_pretrained + G_L * delta_task; a prefix gate G_P(h_in) in (0, 1) scales the virtual prefix key and value matrices (G_P * P_K, G_P * P_V) prepended to multi-head attention while leaving original token keys and values intact; and an adapter gate G_A(h_FN) in (0, 1) scales the post-feedforward bottleneck output via h'_A = G_A * h_A + h_F.