FedMD (Heterogeneous Federated Learning via Public-Dataset Model Distillation)

A model-heterogeneous federated learning framework (Li & Wang 2019, Section 3, Algorithm 1, Figure 2) in which each client k in {1, ..., m} designs an arbitrary, privately held neural network architecture f_k, performs initial transfer learning by pre-training f_k on a shared public dataset D_0 = {x_i^0} and fine-tuning on its private dataset D_k, and then collaborates in each communication round through a four-stage public-logit protocol: (1) Communicate each client's class score (logit) predictions f_k(x_i^0) on D_0 to the server; (2) Aggregate the consensus logit vector f_tilde(x_i^0) = (1/m) sum_{k=1}^m f_k(x_i^0) on the server; (3) Distribute f_tilde(x_i^0) back to all clients; and (4) Digest & Revisit by training each local model f_k to match the consensus logits f_tilde(x_i^0) on D_0 (Digest phase) followed by local fine-tuning epochs on private dataset D_k (Revisit phase).