Home / Curated / Algorithms / FedDF (Ensemble Distillation for Robust Model Fusion in Federated Learning)

FedDF (Ensemble Distillation for Robust Model Fusion in Federated Learning)

Algorithms AIMech:FedDfEnsembleDistillationForRobustModelFusionInFederatedLearning DRAFT

Aliases: FedDF, Federated Distillation Fusion, Ensemble Distillation for Robust Model Fusion in Federated Learning

A server-side ensemble distillation framework for federated model fusion (FedDF, Algorithm 1 for homogeneous models and Algorithm 3 for heterogeneous model prototypes) in which, at each communication round t, the server initializes each student prototype from the coordinate-wise parameter average x_{t,0}^l = (1/|S_t^l|) sum_{k in S_t^l} x_hat_t^k of participating clients in architecture group l, and then executes N_distill steps of knowledge distillation on unlabeled or synthetically generated server mini-batches d by minimizing the Kullback-Leibler divergence KL(sigma((1/|S_t|) sum_{k in S_t} f(x_hat_t^k, d)), sigma(f(x_{t,j}^l, d))) between the softmax of the ensemble-averaged client logits and the student model logits.

Identity Notes & Concept Scope Distinct from coordinate-wise parameter-averaging schemes such as FedAvg, FedProx (AIMech:FedProxProximalHeterogeneousFederatedOptimization), FedNova (AIMech:FedNovaNormalizedAveragingForHeterogeneousFederatedOptimization), and FedOpt (AIMech:FedOptAdaptiveServerOptimizationFedAdagradFedYogiFedAdam), which cannot fuse heterogeneous neural network architectures and suffer functional collapse when averaging heavily drifted non-linear client weights, and from FedMA (AIMech:FedMaFederatedMatchedAveragingViaLayerWiseBayesianNonparametricMatching, which performs layer-wise Hungarian permutation alignment in weight space without unlabeled server-side logit distillation).

Claims & Grounded Evidence

4 Claims

Mechanism #r407-logit-ensemble-kl-distillation-and-heterogeneous-architecture-fusion

In homogeneous FedDF (Algorithm 1), initializing the server student from the uniform parameter average x_{t,0} = (1/|S_t|) sum_{k in S_t} x_hat_t^k and refining it via gradient descent on KL(sigma((1/|S_t|) sum_{k in S_t} f(x_hat_t^k, d)), sigma(f(x_t, d))) tolerates large numbers of local training epochs and learning rates where FedAvg diverges, while heterogeneous FedDF (Algorithm 3) fuses knowledge across p distinct model architectures M^1,...,M^p by distilling the full cross-architecture logit average into each prototype architecture.

supports Source: Ensemble Distillation for Robust Model Fusion in Federated Learning · Locator: Section 3.1-3.2 (Algorithm 1), Section 4.1-4.2 (Figures 3-6, Tables 1-2), and Appendix A (Algorithm 3), pp. 3-8, 16

Mechanism #r407-multi-source-domain-adaptation-ensemble-generalization-bound

Viewing each client k as a local source domain D_k and the distillation distribution as D, Theorem 5.1 (and Theorem D.1) bounds the expected risk of the uniform client ensemble by L_D((1/K) sum_k h_{D_hat_k}) <= L_{D_hat}(h_{D_hat}) + (1/K) sum_k ((1/2) d_{H Delta H}(D_k, D) + lambda_k) + O(sqrt(log(K * tau_H(2m) / delta) / m)), establishing that the ensemble risk is at most the centralized pooled risk plus average H-Delta-H domain discrepancy.

supports Source: Ensemble Distillation for Robust Model Fusion in Federated Learning · Locator: Section 5 Understanding Ensemble Distillation (Theorem 5.1, Figure 7) & Appendix D (Theorem D.1), pp. 8-9, 24-25

Limitation #r407-unlabeled-proxy-data-or-generator-and-server-compute-limitation

FedDF requires access at the server to an unlabeled proxy dataset (out-of-domain or synthetic data from a pretrained generator) and incurs additional server-side forward passes across |S_t| teacher models and N_distill student optimization steps in each communication round.

supports Source: Ensemble Distillation for Robust Model Fusion in Federated Learning · Locator: Section 3.2 Discussion & Section 4.1 Ablation Study of Public Data (Figure 5), pp. 4-7

Application #r407-cifar-imagenet-agnews-sst2-and-heterogeneous-resnet-vgg-shufflenet-application

Evaluated on homogeneous and heterogeneous federated learning across CIFAR-10, CIFAR-100, ImageNet, AG News, and SST2 (fusing ResNet-8, VGG-9, MobileNetV2, and ShuffleNetV2 prototypes, and fine-tuning DistilBERT), outperforming FedAvg, FedProx, FedAvgM, and FedMA under severe Dirichlet non-IID partitions (alpha = 0.01 to 1.0).

supports Source: Ensemble Distillation for Robust Model Fusion in Federated Learning · Locator: Section 4 Empirical Results (Tables 1-3, Figures 3-6), pp. 5-8

⚠️ Curated Limitations & Negative Results

Requires an unlabeled proxy dataset or synthetic data generator at the server to evaluate client and student logits.
Incurs additional server-side compute to run forward passes over participating client models during distillation.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	narrow_match	FedDF is a FederatedLearning model-fusion algorithm that aggregates client models via server-side ensemble logit distillation.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r407-primary-pdf`	Ensemble Distillation for Robust Model Fusion in Federated Learning Tao Lin, Lingjing Kong, Sebastian U. Stich, Martin Jaggi	primary paper full_text	ID: https://proceedings.neurips.cc/paper/2020/hash/18df51b97ccd68128e994804f3eccc87-Abstract.html sha256:1fb3f82ed65c…
`r407-arxiv-abs`	Ensemble Distillation for Robust Model Fusion in Federated Learning (arXiv abstract) Tao Lin, Lingjing Kong, Sebastian U. Stich, Martin Jaggi	primary paper abstract_only	ID: https://arxiv.org/abs/2006.07242 sha256:3eb89e50673e…
`r407-metadata-cr`	Ensemble Distillation for Robust Model Fusion in Federated Learning (Crossref metadata) Tao Lin, Lingjing Kong, Sebastian U. Stich, Martin Jaggi	primary paper abstract_only	ID: https://proceedings.neurips.cc/paper/2020/hash/18df51b97ccd68128e994804f3eccc87-Abstract.html sha256:a2405b3537fe…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T08:10:00Z

Updated: 2026-10-19T08:10:00Z

Source File: data/curated/FedDfEnsembleDistillationForRobustModelFusionInFederatedLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories