AdaSum Adaptive Summation for Distributed Training
Algorithms AIMech:AdaSumAdaptiveSummationForDistributedTraining DRAFT
Aliases: AdaSum, Adaptive Summation for Distributed Training, ADASUMRVH
A hyperparameter-free distributed gradient and model-update reduction operator that replaces naive all-reduce summation across data-parallel workers by recursively combining pairs of layer-wise gradients or optimizer parameter deltas (g_1, g_2) via Adasum(g_1, g_2) = (1 - (g_1^T g_2) / (2 ||g_1||_2^2)) g_1 + (1 - (g_1^T g_2) / (2 ||g_2||_2^2)) g_2, derived by averaging the first-order Taylor sequential SGD emulations of executing minibatch b_1 before b_2 and b_2 before b_1 under a rank-one outer-product Fisher Hessian approximation H_i approx g_i g_i^T and locally optimal step size alpha = 1 / ||g_i||_2^2.
Identity Notes & Concept Scope
Distinct from standard synchronous AllReduce summation/averaging, from delay-compensated asynchronous SGD (DC-ASGD, which uses only diagonal g_i odot g_i and requires a tuned hyperparameter), and from per-layer norm trust-ratio optimizers (LARS, LAMB): AdaSum operates across workers via a two-phase recursive vector-halving collective (ADASUMRVH) after local optimizer steps, reducing to the exact sum g_1 + g_2 when g_1 and g_2 are orthogonal and to the exact average (g_1 + g_2)/2 when g_1 = g_2.
Claims & Grounded Evidence
4 Claims
Mechanism
#c-r623-sequential-emulation-and-orthogonality-interpolation
Approximating the second-step gradient g_2(w_0 - alpha g_1) by its first-order Taylor expansion g_2 - alpha H_2 g_1 with Fisher Hessian H_2 approx g_2 g_2^T and optimal quadratic step alpha = 1 / ||g_2||_2^2 yields w_{1,2} = w_0 - alpha (g_1 + (1 - (g_2^T g_1)/||g_2||_2^2) g_2); averaging both minibatch execution orders (b_1, b_2) and (b_2, b_1) produces Adasum(g_1, g_2), which equals g_1 + g_2 when g_1^T g_2 = 0 (orthogonal) and (g_1 + g_2)/2 when g_1 = g_2 (parallel).
Mechanism
#c-r623-pseudogradient-angle-and-norm-bounds
For independent gradient samples a, b ~ X with mean r = E[X], E[Adasum(a, b)] = (2I - E[a a^T / ||a||_2^2]) r satisfies ||E[X]||_2 <= ||E[Adasum(a, b)]||_2 <= 2 ||E[X]||_2 (Lemma A.3) and forms an angle eta with E[X] satisfying cos(eta) >= min_gamma (2 - cos^2(gamma)) / sqrt(4 - 3 cos^2(gamma)) = 2 sqrt(2) / 3 approx 0.9428 > 0.942 (eta <= 0.108 pi, Lemma A.2), establishing pseudogradient convergence across binary-tree reductions (Theorem A.4).
Application
#c-r623-adasumrvh-and-large-batch-scaling
Implemented via a two-phase recursive vector-halving collective (ADASUMRVH, Algorithm 1) applied per layer to post-optimizer parameter deltas with FP64 dot-product accumulation, AdaSum scales Momentum-SGD on ResNet-50 and Adam on BERT-Large from 16K to 64K effective batch size and LAMB on BERT-Large from 64K to 128K effective batch size while converging in up to 30% fewer steps at 64K.
Limitation
#c-r623-non-streaming-dot-product-and-parallel-gradient-limitation
Because Adasum(a, b) requires global inner products a^T b and squared norms ||a||_2^2, ||b||_2^2 across full layer tensors before scaling, it cannot be executed as a single-pass elementwise MPI reduction and requires power-of-two recursive vector-halving with extra subgroup dot-product allreduces; moreover, if all N worker gradients are strictly parallel, its convergence rate reduces to 1/N of sequential SGD.
⚠️ Curated Limitations & Negative Results
- Cannot be implemented as a streaming elementwise MPI AllReduce operator because each tree level requires a subgroup allreduce of partial dot products [a . b, a . a, b . b] across full layer slices before applying the scaling factors.
- Requires double-precision (FP64) accumulation for inner products and squared norms plus dynamic loss scaling on effective gradients when training in FP16 to prevent numerical underflow/overflow.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | AdaSum is a distributed gradient/model-update combination algorithm for scaling data-parallel deep neural network training to large effective batch sizes. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r623-primary-pdf
|
Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Olli Saarikivi, Tianju Xu, Vadim Eksarevskiy, Jaliya Ekanayake, Emad Barsoum
|
primary paper
full_text
|
ID: https://dblp.org/rec/conf/mlsys/MalekiMMSXEEB21
sha256:632cf2c1a00c…
|
r623-arxiv-abs
|
Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Olli Saarikivi, Tianju Xu, Vadim Eksarevskiy, Jaliya Ekanayake, Emad Barsoum
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2006.02924
sha256:4dcddcf412b1…
|
r623-metadata-cr
|
Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Olli Saarikivi, Tianju Xu, Vadim Eksarevskiy, Jaliya Ekanayake, Emad Barsoum
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/conf/mlsys/MalekiMMSXEEB21
sha256:9fb0228fca12…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T11:46:00Z
Updated: 2026-10-19T11:46:00Z
Source File:
data/curated/AdaSumAdaptiveSummationForDistributedTraining.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories