Diff Pruning

A parameter-efficient transfer learning algorithm that formulates task-specific fine-tuning as learning a sparse difference ('diff') vector delta_tau added to frozen shared pre-trained parameters theta via theta_tau = theta + delta_tau. To learn which coordinates to modify during training, Diff Pruning decomposes delta_tau = z_tau (*) w_tau using a stretched Hard-Concrete gate z_tau = min(1, max(0, s_tau * (r - l) + l)) with s_tau = sigma(log u - log(1 - u) + alpha_tau), u ~ U(0, 1), l < 0, and r > 1, and optimizes a closed-form differentiable expected L0-norm penalty E[||delta_tau||_0] = sum_{i=1}^d sigma(alpha_{tau,i} - log(-l/r)). In its Structured Diff Pruning extension, parameters are partitioned into G matrix/bias groups {g(1), ..., g(G)} with per-group gates z_tau^j so delta_{tau,i}^j = z_{tau,i} * z_tau^j * w_{tau,i} and E[||delta_tau||_0] = sum_{j=1}^G sum_{i in g(j)} sigma(alpha_{tau,i} - log(-l/r)) * sigma(alpha_tau^j - log(-l/r)). After training, exact budget control (e.g., t% = 0.5% nonzeros) is achieved by projecting delta_tau onto the target L0-ball via magnitude pruning on delta_tau followed by fixed-mask fine-tuning.