Fed-ET (Heterogeneous Ensemble Knowledge Transfer with Confidence-Weighted Consensus and Diversity Regularization)

A bi-directional ensemble knowledge transfer framework for model-heterogeneous federated learning (Cho, Manoel, Joshi, Sim, & Dimitriadis 2022, Section 3, Equations 1-13, Algorithm 1, Theorem 1) that trains a large server model w_bar in R^n (with representation layer h_bar in R^u) using a hashmap M = {1: w_bar_1, ..., U: w_bar_U} of smaller heterogeneous client architectures w_bar_i in R^{n_i} (n >> n_i, each equipped with a matched representation layer h_bar_i in R^u) and an unlabeled server-side public dataset P without requiring client access to P. Each round t executes three steps: (1) selected clients k in S^{(t,0)} train their designated small models w_k^{(t,0)} = M[M(k)] on private data B_k (Equation 1) and the server pre-conditions its representation layer via h_bar^{(t,0)} = (1/m) sum_{k in S^{(t,0)}} h_k^{(t,tau)}; (2) on each unlabeled sample x in P, the server computes each client's intra-logit probability variance sigma_s^2(w_k^{(t,tau)}, x) = Var(s(w_k^{(t,tau)}, x)) across the N classes, forms the variance-weighted consensus logit s^{(t,tau)}(x) = sum_{k in S^{(t,0)}} alpha_k^{(t,tau)}(x) s(w_k^{(t,tau)}, x) with alpha_k^{(t,tau)}(x) = sigma_s^2(w_k^{(t,tau)}, x) / sum_{l in S^{(t,0)}} sigma_s^2(w_l^{(t,tau)}, x) and pseudo-label y_s^{(t,tau)}(x) = argmax s^{(t,tau)}(x) (Equations 2-4), aggregates dissenting clients S_{div}^{(t,0)}(x) = {l in S^{(t,0)} : y_s^{(t,tau)}(x) != argmax s(w_l^{(t,tau)}, x)} into a diversity logit s_{div}^{(t,tau)}(x) = sum_{k in S_{div}^{(t,0)}(x)} alpha_k(x) s(w_k^{(t,tau)}, x) (Equations 6-8), and updates w_bar by minimizing F(w_bar^{(t,0)}) = (1/|P|) sum_{x in P} [l((x, y_s^{(t,tau)}(x)), w_bar^{(t,0)}) + lambda KL(s_{div}^{(t,tau)}(x), s(w_bar^{(t,0)}, x))] (Equations 10-11); and (3) aggregates same-architecture client models M[i] (Equation 12) and transfers the updated server representation layer h_bar^{(t,tau_s)} back to all models in M.