FedAUX (Certainty-Weighted Unsupervised Auxiliary Ensemble Distillation for Federated Learning)

An unlabeled-auxiliary-data federated learning and ensemble distillation framework (Sattler, Korjakow, Rischke, & Samek 2021, Sections 3.2-3.5, Equations 1-10, Algorithm 1, Theorems 1-2) that leverages an unlabeled auxiliary dataset D_aux = D_train U D_distill U D^- in two stages: (1) self-supervised contrastive feature extractor pretraining of h_0 on D_train to initialize client feature extractors in a common representation basin, and (2) certainty-weighted server ensemble distillation on D_distill where each client i solves a strongly convex L2-regularized logistic regression problem J(w, h_0, D_i, D^-) = (1/(|D_i| + |D^-|)) sum_{x in D_i U D^-} log(1 + exp(-t_x <w, h_tilde_0(x)>)) + (lambda/2) ||w||_2^2 separating its private data D_i (t_x = +1) from negative auxiliary data D^- (t_x = -1) (Equation 4), perturbs w_i^* with calibrated Gaussian noise N(0, sigma^2 I) for (epsilon, delta)-differential privacy (Equations 5 & 10, Theorem 2), and weights each client's teacher logits f_i(x) on x in D_distill by its certainty score s_i(x) = (1 + exp(-<w_i^* + rho, h_tilde_0(x)>))^{-1} + xi via A_s(x) = softmax(sum_{i in S_t} s_i(x) f_i(x) / sum_{i in S_t} s_i(x)) (Equations 3 & 6).