Home / Curated / Algorithms / Split-Mix Federated Learning

Split-Mix Federated Learning

Algorithms AIMech:SplitMixHeterogeneousWidthSubnetworkFederatedLearning DRAFT

Aliases: Split-Mix, Split-Mix FL, Split-Mix and Split-Mix+DAT, Split-Mix On-Demand and In-Situ Customizable Federated Learning

A model-heterogeneous federated learning and in-situ customization algorithm (Split-Mix; Hong, Wang, Wang, and Zhou, ICLR 2022) that decomposes a full-width network into M = floor(1 / r) independent base sub-networks W = {w_{i,r}}_{i=1}^M of base width ratio r in (0, 1] initialized with rescaled Kaiming initialization matching the full-width variance. In each round t, client k with width budget R_k in {r, 2r, ..., 1} draws floor(R_k / r) base sub-networks via shuffled cyclic without-replacement sampling (SampleBaseModels, Algorithm 2), trains them independently in parallel on local data (LocalUpdate, Algorithm 3), and returns updates aggregated per base sub-network weighted by participating clients' sample sizes (Algorithm 1 lines 7-9). At deployment, an on-demand R-width model aggregates predictions of K_R = floor(R / r) base sub-networks via F(x; W^T) = (1 / K_R) sum_{i=1}^{K_R} f(x; w_{i,r}^T) (Algorithm 1 line 11), and for joint adversarial robustness customization (Split-Mix+DAT) mixes Dual Batch-Normalization statistics via DBN(x) = (1 - lambda) BN_c(x) + lambda BN_n(x) for lambda in [0, 1] (Eq. 2).

Identity Notes & Concept Scope Distinct from nested width-slimming heterogeneous FL algorithms such as HeteroFL (AIMech:HeteroFlWidthScaledSubnetworkAggregationForHeterogeneousFederatedLearning; Diao et al., ICLR 2021) and FjORD (AIMech:FjordOrderedDropoutAndSelfDistillationForHeterogeneousFederatedLearning; Horvath et al., NeurIPS 2021), which slice nested prefix channel blocks from a single monolithic parameter tensor (causing wider channel blocks to be trained exclusively on wide-budget clients) rather than decomposing the model into M = floor(1/r) modular width-r base sub-networks sampled cyclically across all clients and ensembled on demand at inference time. Distinct from FedRolex (AIMech:FedRolexModelHeterogeneousFederatedLearningWithRollingSubModelExtraction; Alam et al., NeurIPS 2022), which extracts a deterministic rolling channel window from a single global model rather than maintaining M independent width-r base sub-networks with training-free Dual Batch-Normalization mixing.

Claims & Grounded Evidence

4 Claims

Mechanism #split-mix-cyclic-base-subnetwork-sampling-and-ensemble-inference-mechanism

In Split-Mix (Algorithms 1-3), the server initializes M = floor(1 / r) base sub-networks {w_{i,r}}_{i=1}^M of width ratio r using rescaled Kaiming initialization with the x1 full-width fan-in variance, assigns each client k with width budget R_k a subset W_k^t of floor(R_k / r) base sub-networks via shuffled cyclic sampling without replacement (Algorithm 2), aggregates each base model w_{i,r}^{t+1} over all clients k whose W_k^t contains w_{i,r}, and constructs an on-demand R-width predictor at inference by averaging K_R = floor(R / r) base sub-network outputs F(x; W^T) = (1 / K_R) sum_{i=1}^{K_R} f(x; w_{i,r}^T).

supports Source: Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization · Locator: Section 4.1 Customize Model Size via Width-wise Splitting and Mixing, Algorithms 1-3, pp. 4-6

Scope #split-mix-dat-dual-batch-normalization-in-situ-robustness-customization-scope

Split-Mix+DAT (Section 4.2, Eq. 2, p. 6; Algorithm 4, p. 14) equips each base sub-network with Dual Batch-Normalization (DBN) consisting of separate clean (BN_c) and PGD-adversarial (BN_n) normalization branches optimized under local loss (1 - beta) ell(f(x; BN_c), y) + beta ell(f(x + delta; BN_n), y), enabling training-free in-situ trade-off between standard accuracy and adversarial robustness after FL training via layer-wise linear interpolation DBN(x) = (1 - lambda) BN_c(x) + lambda BN_n(x) for lambda in [0, 1].

supports Source: Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization · Locator: Section 4.2 Customize Robustness by Mixing Dual Batch Normalization, Eq. (2), p. 6; Appendix A.2 Algorithm 4, p. 14

Limitation #split-mix-discrete-width-granularity-and-pseudo-residual-overhead-limitation

Split-Mix restricts customizable model widths to discrete integer multiples of the base width ratio r (R in {r, 2r, ..., M r}), requires tuning beta and PGD inner maximization steps on clients capable of adversarial training in Split-Mix+DAT, and incurs M-way parallel forward passes at full-width R = 1 inference (Section 4.1-4.2, pp. 4-6; Appendix B.1, pp. 14-16).

supports Source: Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization · Locator: Section 4.1-4.2, pp. 4-6; Appendix B.1 Customize Model Widths, pp. 14-16

Application #split-mix-domain-shift-and-heterogeneous-budget-image-classification-application

Applied to budget-heterogeneous and domain-heterogeneous federated image classification and in-situ robustness customization on CIFAR-10, DomainNet, and Digits (MNIST, SVHN, USPS, SynthDigits, MNIST-M) using PreActResNet-18, AlexNet, and ConvNet backbones.

supports Source: Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization · Locator: Section 5 Experiments, Figures 4-7, Table 1, pp. 6-9

⚠️ Curated Limitations & Negative Results

Customizable inference widths are quantized to discrete multiples of the base sub-network width ratio r (Section 4.1, pp. 4-5).
Full-width (R = 1) inference evaluates M = floor(1 / r) independent width-r base sub-networks in parallel rather than a single dense weight tensor (Algorithm 1 line 11, p. 5).
In-situ adversarial robustness customization via Split-Mix+DAT requires a non-empty fraction of clients with sufficient compute budget to perform local PGD adversarial training on Dual Batch-Normalization branches (Section 4.2, p. 6; Section 5.2, p. 8).

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	broad_match	Split-Mix is a model-heterogeneous federated learning algorithm with width-wise base sub-network splitting and on-demand mixing.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r506-primary-pdf`	Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization Junyuan Hong, Haotao Wang, Zhangyang Wang, Jiayu Zhou	primary paper full_text	ID: https://dblp.org/rec/conf/iclr/HongWWZ22 sha256:ce80fbc0dde3…
`r506-arxiv-abs`	Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization (arXiv abstract) Junyuan Hong, Haotao Wang, Zhangyang Wang, Jiayu Zhou	primary paper abstract_only	ID: https://arxiv.org/abs/2203.09747 sha256:5de0cf8a9a0d…
`r506-metadata-cr`	Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization (Crossref metadata) Junyuan Hong, Haotao Wang, Zhangyang Wang, Jiayu Zhou	primary paper abstract_only	ID: https://dblp.org/rec/conf/iclr/HongWWZ22 sha256:881de19f952b…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T09:49:00Z

Updated: 2026-10-19T09:49:00Z

Source File: data/curated/SplitMixHeterogeneousWidthSubnetworkFederatedLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories