Home / Curated / Algorithms / FjORD (Fair and Accurate Federated Learning under Heterogeneous Targets with Ordered Dropout)

FjORD (Fair and Accurate Federated Learning under Heterogeneous Targets with Ordered Dropout)

Algorithms AIMech:FjordOrderedDropoutAndSelfDistillationForHeterogeneousFederatedLearning DRAFT

Aliases: FjORD, Federated Learning with Ordered Dropout, Ordered Dropout for Heterogeneous Federated Learning

A system-heterogeneous federated learning method (Horvath, Laskaridis, Almeida, Leontiadis, Venieris, & Lane 2021, Sections 3-4, Equation 1, Theorem 1) that performs Ordered Dropout (OD) by sampling a prefix width rate p ~ D_P up to each client's maximum device capability p_max^i and pruning the rightmost channels to retain only prefix indices {0, ..., ceil(p * K_l) - 1} in each layer l, combines OD with client-side self-distillation from the client's maximal submodel F_{p_max^i} to smaller sampled prefix submodels F_p, and aggregates disjoint width slices w_j = w_{ceil(p_j K)} \ w_{ceil(p_{j-1} K)} on the server over eligible clients {i : p_max^i >= p_j} (Equation 1).

Identity Notes & Concept Scope Distinct from HeteroFL (AIMech:HeteroFlWidthScaledSubnetworkAggregationForHeterogeneousFederatedLearning, Diao et al. 2021), which trains each client only at its single static capability width p_max^i rather than stochastically sampling all prefix widths p <= p_max^i via Ordered Dropout and self-distillation to enforce SVD-ordered representation importance across nested submodels.

Claims & Grounded Evidence

4 Claims

Mechanism #r448-ordered-dropout-svd-best-low-rank-equivalence

Theorem 1 (Section 3.2, p. 4) proves that for a two-layer linear network trained under Ordered Dropout with positive-probability prefix rates p in {j/K}_{j=1}^K, any global optimum orders the hidden units by singular value magnitude so that every prefix submodel of width b in {1, ..., K} equals the exact Eckart-Young-Mirsky best rank-b approximation of the full mapping.

supports Source: FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout · Locator: Section 3.2 Theoretical Analysis (Theorem 1), p. 4 & Appendix A Proof of Theorem 1

Mechanism #r448-self-distillation-and-tier-aware-slice-aggregation-mechanism

During local client training, FjORD uses the soft predictions of the client's largest runnable submodel p_max^i as a self-distillation teacher for smaller sampled prefix submodels p < p_max^i (Section 3.3), and the server updates each incremental parameter slice w_{ceil(p_j K)} \ w_{ceil(p_{j-1} K)} exclusively over clients with capability p_max^i >= p_j (Section 4, Equation 1).

supports Source: FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout · Locator: Section 3.3 Knowledge Distillation & Section 4 FjORD (Equation 1, Algorithm 1), pp. 5-7

Limitation #r448-prefix-width-grid-and-separate-bn-statistics-limitation

Ordered Dropout restricts submodels to a discrete set of prefix width rates P = {p_1, ..., p_M}, requires maintaining separate Batch Normalization layers per width rate p in P, and increases local forward/backward passes when computing both the teacher pass at p_max^i and student pass at p < p_max^i.

supports Source: FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout · Locator: Section 3.1 Ordered Dropout Architecture & Section 3.3 Knowledge Distillation, pp. 3-5

Application #r448-cifar10-femnist-and-shakespeare-multi-tier-deployment-application

Applied to heterogeneous-device federated learning and dynamic test-time width scaling on CIFAR-10 (ResNet-18), FEMNIST (CNN), and Shakespeare (RNN).

supports Source: FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout · Locator: Section 5 Experimental Evaluation (Figures 3-6, Tables 1-2), pp. 7-9

⚠️ Curated Limitations & Negative Results

Self-distillation requires an extra forward pass through the maximal runnable submodel p_max^i on each mini-batch.
Requires separate Batch Normalization layers for each discrete width rate p in P.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	narrow_match	FjORD is a system-heterogeneous federated learning algorithm that trains nested width-pruned submodels via Ordered Dropout and self-distillation.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r448-primary-pdf`	FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I. Venieris, Nicholas D. Lane	primary paper full_text	ID: https://proceedings.neurips.cc/paper/2021/hash/6aed000af86a084f9cb0264161e29dd3-Abstract.html sha256:439d001376bb…
`r448-arxiv-abs`	FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout (arXiv abstract) Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I. Venieris, Nicholas D. Lane	primary paper abstract_only	ID: https://arxiv.org/abs/2102.13451 sha256:7b86acd7039d…
`r448-metadata-cr`	FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout (Crossref metadata) Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I. Venieris, Nicholas D. Lane	primary paper abstract_only	ID: https://proceedings.neurips.cc/paper/2021/hash/6aed000af86a084f9cb0264161e29dd3-Abstract.html sha256:8684136c57ab…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T08:51:00Z

Updated: 2026-10-19T08:51:00Z

Source File: data/curated/FjordOrderedDropoutAndSelfDistillationForHeterogeneousFederatedLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories