Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization (BOFT)

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization (BOFT) is a multiplicative parameter-efficient fine-tuning method that adapts a frozen pretrained weight matrix W^0 in R^{d x n} via the orthogonal transformation z = W^T x = (R(m, b) W^0)^T x, where the orthogonal matrix R(m, b) in O(d) is factored as a product of m sparse butterfly orthogonal components: R(m, b) = prod_{i=1}^m B~^b_{(d, i)}. Each butterfly factor B~^b_{(d, i)} is a permutation of a block-diagonal orthogonal matrix whose 2 x 2 block of (b/2) x (b/2) submatrices is constrained to be orthogonal via the Cayley transform R_block = (I + Q)(I - Q)^{-1} with skew-symmetric Q = -Q^T, initialized at Q = 0 so R(m, b) = I_d at step 0. With block size b >= 2 and m <= log_2(d/b) + 1 levels, BOFT uses (1/2)(b - 1) d m trainable parameters per layer, subsumes block-diagonal OFT at m = 1, and generates dense orthogonal matrices across the full d x d space when m = log_2(d/b) + 1.