FedAlign (Data-Free Final-Block Lipschitz Self-Distillation via Slimmed Sub-Block Transmitting Matrices)

A resource-efficient local self-distillation regularization algorithm for heterogeneous federated learning (Mendieta, Yang, Wang, Lee, Ding, & Chen 2022, Sections 3.4 & 4, Equations 3-7, Algorithm 1) that mitigates client drift by aligning the approximated Lipschitz constants of a neural network's final block L across full width and a uniformly slimmed fractional width omega_S in (0, 1) while reusing intermediate features from block L - 1. During local training on batch (x, y) in D, each client runs a single full-network forward pass f_{theta_{L-1}}, f_{theta_L}, pred = F_theta(x) and one additional partial forward pass of the final block L at reduced channel width omega_S using f_{theta_{L-1}} as input: f_{theta_L^{omega_S}} = F_{theta_L^{omega_S}}(f_{theta_{L-1}}). The client constructs feature-map transmitting matrices X_F = (f_{theta_{L-1}})^T f_{theta_L} and X_S = (f_{theta_{L-1}})^T f_{theta_L^{omega_S}} (Equations 6-7), approximates their spectral norms K_F = ||X_F||_{SN} and K_S = ||X_S||_{SN} via Power Iteration, and minimizes the local objective L_{FA} = L_{CE}(F_theta(x), y) + mu L_{Lip}(K_S, K_F) (Equation 5), where L_{Lip}(K_S, K_F) is the mean squared error between K_S and K_F.