Home / Curated / Algorithms / SlowMo (Slow Momentum for Communication-Efficient Distributed SGD)

SlowMo (Slow Momentum for Communication-Efficient Distributed SGD)

Algorithms AIMech:SlowMoSlowMomentumForCommunicationEfficientDistributedSgd DRAFT

Aliases: SlowMo, Slow Momentum for Communication-Efficient Distributed SGD, Outer Slow-Momentum Wrapper for Local and Decentralized SGD

A general two-timescale distributed optimization wrapper (Wang et al. 2020, Section 2, Algorithm 1) in which m workers independently execute tau local steps of an arbitrary base optimizer x_{t,k+1}^{(i)} = x_{t,k}^{(i)} - gamma_t d_{t,k}^{(i)} (such as Local SGD, local momentum, or decentralized SGP/OSGP) starting from x_{t,0}^{(i)} = x_{t,0}, compute the exact global average x_{t,tau} = (1/m) sum_{i=1}^m x_{t,tau}^{(i)} every tau steps via AllReduce, and update the outer model via a slow momentum buffer u_{t+1} = beta u_t + (1 / gamma_t)(x_{t,0} - x_{t,tau}), x_{t+1,0} = x_{t,0} - alpha gamma_t u_{t+1} with slow learning rate alpha > 0 and slow momentum factor beta in [0, 1).

Identity Notes & Concept Scope Distinct from standard Local SGD (AIMech:LocalSgdPeriodicAveragingParallelStochasticGradientDescent, Stich 2019), which corresponds to the memoryless special case alpha = 1, beta = 0 without outer slow momentum u_t, and from FedOpt (AIMech:FedOptAdaptiveServerOptimizationFedAdagradFedYogiFedAdam, Reddi et al. 2021), which focuses on adaptive second-moment server preconditioners (FedAdagrad, FedYogi, FedAdam) under partial client participation rather than wrapping decentralized gossip/SGP and local base optimizers with outer Polyak momentum and proving BMUF/Lookahead non-convex convergence.

Claims & Grounded Evidence

4 Claims

Mechanism #r426-auxiliary-extrapolated-sequence-and-nonconvex-convergence-theorem

Defining the extrapolated auxiliary sequence y_{t,0} = x_{t,0} + (beta / (1 - beta)) (x_{t,0} - x_{t-1,0}) converts the outer momentum recurrence into the exact step y_{t+1,0} - y_{t,0} = -gamma_eff sum_{k=0}^{tau-1} (1/m) sum_{i=1}^m d_{t,k}^{(i)} with effective learning rate gamma_eff = alpha gamma / (1 - beta) (Appendix D.1, Eqs. 16-22), proving smooth non-convex convergence at rate O(1 / sqrt(m T tau)) whenever the base optimizer's local drift and bias are bounded (Section 5, Theorem 1).

supports Source: SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum · Locator: Section 5 Convergence Analysis (Theorem 1, Corollaries 1-2) & Appendix D.1-D.3 (Equations 16-22, Proof of Theorem 1)

Mechanism #r426-unification-of-bmuf-and-lookahead-nonconvex-guarantees

Specializing SlowMo's base optimizer and hyperparameters (Section 3, Table 1) provides the first smooth non-convex convergence proofs for Blockwise Model-Update Filtering (BMUF, beta > 0 with Local SGD base, Corollary 1) and Lookahead (m = 1, beta = 0, alpha in (0, 1), Corollary 2), while wrapping Overlap-SGP (OSGP-SlowMo) reduces steady-state validation loss without increasing per-step communication latency.

supports Source: SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum · Locator: Section 3 Relation to Existing Methods (Table 1) & Section 5 Corollaries 1-2

Limitation #r426-periodic-exact-allreduce-and-outer-hyperparameter-tuning-limitation

Requires an exact AllReduce synchronization across all m workers every tau steps to compute x_{t,tau} and introduces two additional outer hyperparameters (slow learning rate alpha and slow momentum beta).

supports Source: SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum · Locator: Section 2-4 Algorithm 1 & Hyperparameter Sensitivity Analysis

Application #r426-imagenet-resnet50-and-wmt16-transformer-distributed-training-application

Applied to distributed training of ResNet-50 on ImageNet and Transformer neural machine translation models on WMT'16 En-De across up to 64 GPUs, consistently improving validation accuracy and training speed over Local SGD, BMUF, and SGP.

supports Source: SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum · Locator: Section 4 Empirical Results (Tables 2-3, Figures 1-4), pp. 4-7

⚠️ Curated Limitations & Negative Results

Maintains an additional O(d) slow-momentum state buffer u_t across outer synchronization epochs.
Requires blocking AllReduce synchronization every tau local iterations.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	related_match	SlowMo wraps multi-step local SGD and decentralized optimization with outer server/all-reduce momentum, directly generalizing federated momentum.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r426-primary-pdf`	SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat	primary paper full_text	ID: https://openreview.net/forum?id=SkxJ8REYPH sha256:dd1de8b49424…
`r426-arxiv-abs`	SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (arXiv abstract) Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat	primary paper abstract_only	ID: https://arxiv.org/abs/1910.00643 sha256:1495c563fc72…
`r426-metadata-cr`	SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (Crossref metadata) Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat	primary paper abstract_only	ID: https://openreview.net/forum?id=SkxJ8REYPH sha256:ed85253e33dc…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T08:29:00Z

Updated: 2026-10-19T08:29:00Z

Source File: data/curated/SlowMoSlowMomentumForCommunicationEfficientDistributedSgd.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories