FedRolex (Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction)

A partial-training (PT) model-heterogeneous federated learning algorithm (Alam, Liu, Yan, & Zhang 2022, Section 3, Equations 3-6, Algorithm 1, Appendix A.1 Lemmas 1-2, Appendix A.2 Equation 10) that trains a global server model theta (which can be larger than the largest client model) across resource-heterogeneous clients n in {1, ..., N} with model capacity proportions beta_n in (0, 1] without requiring public data and while preserving compatibility with secure aggregation. In communication round j, for each layer i of the global model with K_i total nodes/kernels and rolling offset j_hat = j mod K_i, the server extracts client n's sub-model node index set via a deterministic circularly advancing rolling window S_{n,i}^{(j)} = {j_hat, j_hat + 1, ..., j_hat + floor(beta_n K_i) - 1} if j_hat + floor(beta_n K_i) <= K_i, and {j_hat, ..., K_i - 1} U {0, ..., j_hat + floor(beta_n K_i) - 1 - K_i} otherwise (Equation 4), broadcasts the sub-model theta_{n, S_{n,i}^{(j)}}^{(j)} for local SGD training on D_n (Equation 3), and updates each global parameter [i, k] by selective averaging theta_{[i,k]}^{(j+1)} = (1 / sum_{m in M_k} p_m) sum_{m in M_k} p_m theta_{m,[i,k]}^{(j)} over the active clients M_k subset of M that updated parameter [i, k] in round j (Equation 10).