FLoRA: Stacking-Based Federated Fine-Tuning with Heterogeneous Low-Rank Adaptations

A noise-free, rank-heterogeneous federated parameter-efficient fine-tuning algorithm (FLoRA; Wang, Shen, He, Sun, Wang, Lyu, and Li, NeurIPS 2024) that enables K distributed clients with heterogeneous LoRA ranks {r_k}_{k=0}^K to collaboratively fine-tune a pre-trained model W in R^{m x n}. In each communication round, client k initializes and trains local low-rank adapter matrices A_k in R^{r_k x n} and B_k in R^{m x r_k}; instead of independently averaging A_k and B_k (which introduces cross-client error sum_{i != j} p_i p_j B_i A_j and squared diagonal weights p_k^2, Eq. 6), the server constructs global stacked adapters by vertically concatenating scaled A_k blocks A = p_0 A_0 (+) p_1 A_1 (+) ... (+) p_K A_K in R^{(sum_{k=0}^K r_k) x n} and horizontally concatenating B_k blocks B = B_0 (+) B_1 (+) ... (+) B_K in R^{m x (sum_{k=0}^K r_k)} (Eqs. 9-12) (optionally permuting rank-1 sub-modules at random for cross-client privacy), satisfying the exact identity B A = sum_{k=0}^K p_k B_k A_k before redistributing (B, A) to update local base models W <- W + B A.