FedAMP (Federated Attentive Message Passing for Personalized Cross-Silo Federated Learning)

A personalized cross-silo federated learning algorithm (Huang et al. 2021, Sections 3-6, Definition 1, Equations 1-8, Algorithm 1, Theorems 1-2) that jointly optimizes client models W = [w_1, ..., w_m] under a pairwise attention-inducing regularizer min_W G(W) = sum_{i=1}^m F_i(w_i) + lambda sum_{1 <= i < j <= m} A(||w_i - w_j||^2) where A : [0, infty) -> R is a concave, strictly increasing attention-inducing function with A(0) = 0 and bounded right derivative lim_{t -> 0+} A'(t) < infty (such as A(t) = 1 - exp(-t / sigma), Equations 1-2), alternating between (1) a server-side attentive message passing step U^k = W^{k-1} - alpha_k nabla A(W^{k-1}) that forms each client's personalized cloud model u_i^k = sum_{j=1}^m xi_{i,j} w_j^{k-1} with pairwise attention weights xi_{i,j} = alpha_k A'(||w_i^{k-1} - w_j^{k-1}||^2) for j != i and self-weight xi_{i,i} = 1 - sum_{j != i} xi_{i,j} (Equations 3, 5, 7) and (2) client-side proximal point optimization w_i^k = argmin_w {F_i(w) + (lambda / (2 alpha_k)) ||w - u_i^k||^2} (Equations 4, 6), together with the cosine-similarity variant HeurFedAMP (Equation 8) for high-dimensional deep networks.