Home / Curated / Algorithms / 1-bit Adam (Two-Stage Preconditioner-Frozen Error-Compensated Adam)

1-bit Adam (Two-Stage Preconditioner-Frozen Error-Compensated Adam)

Algorithms AIMech:OneBitAdamPreconditionerFreezeCommunicationCompression DRAFT

Aliases: 1-bit Adam, One-Bit Adam, Preconditioner-Frozen Error-Compensated 1-Bit Adam

A two-stage communication-compressed adaptive optimization algorithm (Tang et al. 2021, Section 4, Algorithm 1) that runs uncompressed Adam for a warm-up phase of T_w steps (typically 10-25% of training) until the coordinate-wise second-moment preconditioner v_t = beta_2 v_{t-1} + (1 - beta_2) g_t^2 stabilizes, then freezes V = diag(sqrt(v_{T_w}) + epsilon) and executes bidirectional error-compensated 1-bit compression on the first-order momentum m_t = beta_1 m_{t-1} + (1 - beta_1) g_t followed by the linearly preconditioned update x_t = x_{t-1} - gamma V^{-1} m_hat_t.

Identity Notes & Concept Scope Distinct from linear error-compensated SGD methods such as DoubleSqueeze (AIMech:DoubleSqueezeDoublePassErrorCompensatedCompression) and Sparsified SGD with Error Memory (AIMech:SparsifiedSgdWithErrorMemoryForKContractiveGradientCompression), which lack adaptive coordinate-wise preconditioning V^{-1}, and from server-adaptive FedOpt (AIMech:FedOptAdaptiveServerOptimizationFedAdagradFedYogiFedAdam), which updates adaptive moments on the server after uncompressed pseudo-gradient aggregation rather than freezing Adam's second moment to restore linear error-feedback cancellation.

Claims & Grounded Evidence

4 Claims

Mechanism #r418-nonlinear-variance-error-feedback-incompatibility-and-preconditioner-freeze-telescoping

Because Adam's update V_t^{-1} m_t depends non-linearly on past gradients through v_t = beta_2 v_{t-1} + (1 - beta_2) g_t^2 (Eq. 1), time-varying V_t^{-1} fails to commute with linear error-feedback compensation and leaves a non-telescoping distortion sum_{t} (V_{t+1}^{-1} - V_t^{-1}) delta_t; freezing V = diag(sqrt(v_{T_w}) + epsilon) after warm-up step T_w makes the preconditioned update linear in m_t, restoring exact error cancellation and proving the asymptotic non-convex convergence rate O(||V^{-1/2}||_Fro^2 sigma^2 / sqrt(n T)) matching Adam (Section 4, Theorem 1, Corollary 1).

supports Source: 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed · Locator: Section 3 (Eq. 1), Section 4.1-4.3 (Algorithm 1, Theorem 1, Corollary 1) & Supplementary Material (Proof of Theorem 1)

Mechanism #r418-custom-compressed-allreduce-gather-broadcast-pipeline

Replacing standard ring AllReduce (which cannot directly average 1-bit sign vectors without decompressing at every hop) with a two-phase compressed gather-then-broadcast collective equipped with worker error memory delta_t^{(i)} and server shard error memory epsilon_t reduces inter-node communication volume by up to 97% during the compression stage (Section 5, Figure 4).

supports Source: 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed · Locator: Section 5 System Implementation (Figure 4) & Section 6 Experiments

Limitation #r418-uncompressed-warmup-phase-and-variance-stabilization-limitation

Requires an initial uncompressed Adam warm-up phase of T_w steps (e.g., 13-25% of total steps) before compression can be enabled, and assumes that coordinate second moments v_t reach a quasi-stationary regime by step T_w.

supports Source: 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed · Locator: Section 3 Empirical Observation (Figures 1-3) & Section 6 Warm-up Schedule

Application #r418-bert-large-gpt2-and-sota-language-model-pretraining-application

Applied to distributed pre-training and fine-tuning of BERT-Base, BERT-Large, SQuAD, GLUE, and GPT-2, achieving up to 5.0x fewer communication bits and 3.3x end-to-end training speedup while matching uncompressed Adam accuracy.

supports Source: 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed · Locator: Section 6 Experiments (Tables 1-3, Figures 5-9), pp. 10123-10127

⚠️ Curated Limitations & Negative Results

Cannot compress communication during the initial warm-up phase T_w while the second-moment preconditioner v_t is still evolving rapidly.
Requires storing both frozen preconditioner states v_{T_w} and bidirectional error-compensation buffers in GPU memory.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	related_match	1-bit Adam is a communication-compressed adaptive distributed training algorithm closely related to federated and distributed optimization.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r418-primary-pdf`	1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He	primary paper full_text	ID: https://proceedings.mlr.press/v139/tang21a.html sha256:a66411d7f6f7…
`r418-arxiv-abs`	1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed (arXiv abstract) Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He	primary paper abstract_only	ID: https://arxiv.org/abs/2102.02888 sha256:91c4df3c8aac…
`r418-metadata-cr`	1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed (Crossref metadata) Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He	primary paper abstract_only	ID: https://proceedings.mlr.press/v139/tang21a.html sha256:00f10da29126…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T08:21:00Z

Updated: 2026-10-19T08:21:00Z

Source File: data/curated/OneBitAdamPreconditionerFreezeCommunicationCompression.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories