Home / Curated / Algorithms / HeteroFL (Computation and Communication Efficient Federated Learning for Heterogeneous Clients)

HeteroFL (Computation and Communication Efficient Federated Learning for Heterogeneous Clients)

Algorithms AIMech:HeteroFlWidthScaledSubnetworkAggregationForHeterogeneousFederatedLearning DRAFT

Aliases: HeteroFL, Heterogeneous Federated Learning via Width-Scaled Subnetworks

A system-heterogeneous federated learning framework (Diao, Ding, & Tarokh 2021, Section 3, Equations 1-4, Algorithm 1) that allocates nested top-left width-scaled submatrices W_l^p in R^{d_l^p x k_l^p} with d_l^p = r^{p-1} d_g, k_l^p = r^{p-1} k_g (parameter ratio |W_l^p| / |W_g| = r^{2(p-1)}) from a single global model W_g to clients of computation complexity level p in {1, ..., P}, stabilizes cross-width training via a 1/r^{p-1} Scaler module and static Batch Normalization (sBN), and updates each disjoint parameter partition W_g^{p-1, t+1} \ W_g^{p, t+1} by averaging exclusively over the m - m_{t+1}^{(p)} clients capable of training level p-1 (Equations 2-4).

Identity Notes & Concept Scope Distinct from FjORD (AIMech:FjordOrderedDropoutAndSelfDistillationForHeterogeneousFederatedLearning, Horvath et al. 2021), which randomly samples prefix widths p ~ D_P during local client SGD (Ordered Dropout) and applies self-distillation from each client's maximal submodel rather than training a single fixed submodel level p_i per client round with a 1/r^{p-1} activation Scaler and static Batch Normalization.

Claims & Grounded Evidence

4 Claims

Mechanism #r447-nested-submatrix-partitioned-aggregation-mechanism

By nesting local weight matrices W_l^P subset W_l^{P-1} subset ... subset W_l^1 = W_g at the top-left corner and averaging each disjoint difference set W_g^{p-1, t+1} \ W_g^{p, t+1} only over the m - m_{t+1}^{(p)} clients with complexity index p_i <= p - 1 (Equations 2-4), HeteroFL enables smallest submodels (W_g^P) to benefit from all m clients while higher-capacity outer blocks are updated without zero-padding bias.

supports Source: HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients · Locator: Section 3.1 Heterogeneous Models (Equations 1-4, Figure 1), pp. 3-5

Mechanism #r447-static-batch-norm-scaler-and-masked-cross-entropy-mechanism

To prevent activation magnitude shift across width levels r^{p-1} and running-statistic divergence across non-IID submodels, HeteroFL introduces a parameter-free Scaler module multiplying activations by 1/r^{p-1}, replaces running-moment Batch Normalization during training with Static Batch Normalization (sBN) followed by post-training local moment recalculation, and applies Masked Cross-Entropy on the output layer under non-IID labels (Sections 3.2-3.4).

supports Source: HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients · Locator: Section 3.2 Static Batch Normalization, Section 3.3 Scaler & Section 3.4 Masked Cross-Entropy, pp. 4-6

Limitation #r447-fixed-per-round-width-assignment-and-high-tier-sample-scarcity-limitation

Because each client trains only its fixed assigned width p_i during a round without random prefix dropout across smaller widths, high-capacity parameter blocks W_g^1 \ W_g^2 receive updates only from the highest-tier clients and are not explicitly regularized to preserve submodel accuracy when evaluated at smaller prefix widths.

supports Source: HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients · Locator: Section 3.1 Equations 2-4 & Section 4 Experiments (Tables 2-5), pp. 4-8

Application #r447-mnist-cifar10-and-wikitext2-heterogeneous-federated-application

Applied to computation- and communication-constrained federated learning across 5 exponentially scaled complexity tiers (a-e) on MNIST, CIFAR-10 (ResNet-18), and WikiText-2 (Transformer).

supports Source: HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients · Locator: Section 4 Experiments (Tables 1-5, Figures 2-3), pp. 6-9

⚠️ Curated Limitations & Negative Results

Outer weight partitions W_g^1 \ W_g^2 are trained only by high-capacity clients, which can degrade full-model generalization when high-tier clients hold skewed data.
Requires a post-training communication pass to query local Batch Normalization statistics across clients.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	narrow_match	HeteroFL is a federated learning method that trains nested width-scaled submodels across clients with heterogeneous compute and communication capacities.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r447-primary-pdf`	HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients Enmao Diao, Jie Ding, Vahid Tarokh	primary paper full_text	ID: https://openreview.net/forum?id=TNkPBBYFkXg sha256:93c0c26c9bdc…
`r447-arxiv-abs`	HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients (arXiv abstract) Enmao Diao, Jie Ding, Vahid Tarokh	primary paper abstract_only	ID: https://arxiv.org/abs/2010.01264 sha256:17861a807beb…
`r447-metadata-cr`	HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients (Crossref metadata) Enmao Diao, Jie Ding, Vahid Tarokh	primary paper abstract_only	ID: https://openreview.net/forum?id=TNkPBBYFkXg sha256:2e45364f1361…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T08:50:00Z

Updated: 2026-10-19T08:50:00Z

Source File: data/curated/HeteroFlWidthScaledSubnetworkAggregationForHeterogeneousFederatedLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories