FedFomo (Personalized Federated Learning with First-Order Model Optimization)

A personalized federated learning algorithm (Zhang et al. 2021, Section 3.1 & Appendix A.1, Equations 1-13, Algorithm 1) in which each client c_i maintains a personalized model theta_i^{l(t)} and computes client-specific model aggregation weights w = <w_1, ..., w_N> over M downloaded peer models {theta_n^{l(t)}}_{n=1}^N by taking a first-order Taylor expansion along the convex interpolation path phi_n(w_n) = w_n theta_n^{l(t)} + (1 - w_n) theta_i^{l(t-1)} on a local validation split D_i^{val} to obtain closed-form directional improvement scores w_n = (L_i(theta_i^{l(t-1)}) - L_i(theta_n^{l(t)})) / ||theta_n^{l(t)} - theta_i^{l(t-1)}|| (Equations 3 & 12), ReLU-clipping and normalizing w_n^* = max(w_n, 0) / sum_{m=1}^N max(w_m, 0) (Equation 13) to update theta_i^{l(t)} <- theta_i^{l(t-1)} + sum_{n in [N]} w_n^* (theta_n^{l(t)} - theta_i^{l(t-1)}), and updating a server-side K x K client-to-client affinity matrix P via p_{i,n} <- p_{i,n} + w_n for epsilon-greedy peer model selection.