HyperGrid (Grid-wise Decomposable Hyper Projections for Multi-Task Transformers)

Given first-token (task-prefix) pooled representation X in R^{d_m} and the second positional FFN weight matrix W in R^{d_m x d_f}, HyperGrid segments W into (d_m / d_r) x (d_f / d_c) blocks by computing a d_r x d_c gate via sigmoid outer products—Local-Local H_{L^2}(X) = psi(sigma((L_r X)(L_c X)^T)) odot W (Eq. 3), Local-Global H_{LG}(X) = psi(sigma((L_r X) G_c^T)) odot W (Eq. 4), or Global-Local H_{GL}(X) = psi(sigma(G_r (L_c X)^T)) odot W (Eq. 5)—where psi(.) repeats the d_r x d_c matrix d_m / d_r times along rows and d_f / d_c times along columns, and combines the gated transformation with the pretrained FFN layer via residual addition Y_i = H_i(X_{i-1}, W_i) + W_i(X_{i-1}) (Eq. 6).

At each Transformer layer, HyperGrid adds d_m d_r + d_c parameters in the LG setting, d_r + d_f d_c parameters in the GL setting, and d_m d_r + d_f d_c parameters in the L^2 setting (Section 3.2, p. 5), and gating FFN weights prior to the residual output consistently outperforms post-ReLU Output Gating Y = max(W x + b, 0) odot (sigma(U X) 1^T) (Eq. 7), which suffers a net accuracy loss on SuperGLUE (-0.2% to -1.2% in Table 3).

Co-trained as a single unified multi-task model across all 16 GLUE and SuperGLUE tasks, HyperGrid improves T5-Base SuperGLUE dev score from 73.55 to 75.30 (+2.7%) and T5-Large SuperGLUE dev score from 80.04 to 81.56 (+1.9%), and on the official GLUE and SuperGLUE test servers a single T5-11B HyperGrid (LG, 32 x 128) model achieves 89.4 on GLUE and 87.7 on SuperGLUE using 16x fewer total parameters (11B vs. 176B) than single-task fine-tuned T5-11B.

HyperGrid updates all backbone Transformer parameters jointly with the HyperGrid projections during multi-task fine-tuning (requiring 64 to 128 TPU v3 chips for Large/3B/11B models) and exhibits sensitivity to the choice of fan-in d_r and fan-out d_c grid dimensions, with intermediate fan-out sizes around d_c = 64-128 underperforming both fine-grained (d_c = 32) and coarse (d_c = 256) grid partitions.