Home / Curated / Algorithms / PowerGossip Low-Rank Communication Compression for Decentralized Deep Learning

PowerGossip Low-Rank Communication Compression for Decentralized Deep Learning

Algorithms AIMech:PowerGossipLowRankCommunicationCompressionForDecentralizedDeepLearning DRAFT

Aliases: PowerGossip, PowerGossip Low-Rank Compressed Decentralized SGD, Practical Low-Rank Communication Compression in Decentralized Deep Learning

A hyperparameter-free compressed gossip and decentralized stochastic optimization algorithm (PowerGossip) that directly compresses pairwise model parameter matrix differences X_j - X_i in R^(p x q) across connected neighbors (i, j) using linear rank-1 projection operators C_{v_{ij}}(X_j - X_i) = (X_j v_{ij} - X_i v_{ij}) v_{ij}^T updated via alternating power-iteration steps v^(2k+1) = D^(2k) v^(2k) / ||D^(2k) v^(2k)||_2 and v^(2k) = (D^(2k-1))^T v^(2k-1) / ||(D^(2k-1))^T v^(2k-1)||_2 on D^(k) = X_j^(k) - X_i^(k), enabling distributed vector exchange of p_j = X_j v_hat and p_i = X_i v_hat without communicating full p x q matrices or tuning a separate consensus stepsize.

Identity Notes & Concept Scope Introduced by Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi (NeurIPS 2020, Vol. 33, pp. 14171-14181; arXiv:2008.01425). Distinct from PowerSGD (Vogels, Karimireddy, & Jaggi, NeurIPS 2019), which performs centralized all-reduce low-rank approximation of error-compensated gradients rather than pairwise edge-wise model-difference compression; and distinct from Choco-SGD / Choco-Gossip (Koloskova et al., ICML 2019) and DeepSqueeze (Tang et al., 2019), which maintain full-model neighbor replicas or error buffers and require tuning an additional consensus stepsize gamma.

Claims & Grounded Evidence

4 Claims

Mechanism #powergossip-distributed-linear-projection-and-power-iteration-mechanism

Exploiting the linearity of matrix-vector multiplication, connected nodes i and j maintain a shared projection vector v_{ij} = -v_{ji}, locally compute vectors p_j = X_j v_hat and p_i = X_i v_hat for v_hat = v_{ij} / ||v_{ij}||_2, exchange p_j and p_i to form the rank-1 update Q_hat = (p_j - p_i) v_hat^T = C_{v_hat}(X_j - X_i), and update v_{ij} <- p_j - p_i (alternating between X and X^T across s power-iteration steps) while preserving the exact global average X_bar^(t) = X_bar^(0) (Lemma 1).

supports Source: Practical Low-Rank Communication Compression in Decentralized Deep Learning · Locator: Section 4 Equations (2)-(3), Algorithms 1-2, & Appendix A Lemma 1, pp. 3-4, 12

Scope #powergossip-q-linear-consensus-and-asymptotically-independent-sgd-scope

For any delta-approximate unbiased linear projection operator C satisfying C(X+Y)=C(X)+C(Y), C(C(X))=C(X), and E[C(X)]=delta*X (such as random rank-k subspace projection with delta = k/q) over a mixing matrix W with spectral gap rho = 1 - lambda_2^2 > 0, compressed consensus achieves a parameter-free Q-linear rate (1/N) sum_{i=1}^N E||X_i^(t) - X_bar^(0)||_F^2 <= (1 - rho*delta) (1/N) sum_{i=1}^N ||X_i^(t-1) - X_bar^(0)||_F^2 (Theorem I), and compressed stochastic optimization reaches epsilon-accuracy in O(L*sigma^2 / (n*epsilon^2) + sqrt(L)*(zeta+sigma) / (rho*delta*epsilon^(3/2)) + L / (rho*delta*epsilon)) non-convex rounds (Theorem II).

supports Source: Practical Low-Rank Communication Compression in Decentralized Deep Learning · Locator: Section 5.1 Assumptions (A4)-(A5) & Section 5.2 Theorems I-II, pp. 5-6

Limitation #powergossip-peaky-spectrum-and-per-edge-vector-state-limitation

Low-rank power-iteration compression outperforms random projections only when inter-worker parameter differences exhibit a peaky singular-value spectrum (whereas on unstructured i.i.d. Gaussian matrices no compressed gossip outperforms uncompressed gossip per bit), and each node must store previous projection vectors v_{ij} scaling linearly with its number of graph neighbors.

supports Source: Practical Low-Rank Communication Compression in Decentralized Deep Learning · Locator: Section 4.1 Properties, Section 6 Figure 1, & Appendix E.2, pp. 4, 7, 16

Application #powergossip-decentralized-cifar-resnet20-and-wikitext2-lstm-application

Applied to decentralized training of ResNet-20 on CIFAR-10 (8-worker ring, 34x-57x compression) and 3-layer LSTMs on WikiText-2 language modeling (16-worker ring, 35x-122x compression) using the exact centralized SGD learning rate without tuning any compression hyperparameters.

supports Source: Practical Low-Rank Communication Compression in Decentralized Deep Learning · Locator: Section 6 Experimental analysis, Tables 1-2, pp. 7-8

⚠️ Curated Limitations & Negative Results

Requires storing per-edge projection vectors v_{ij} for each neighbor connection.
Theoretical convergence guarantees (Theorems I-II) are proven for unbiased linear projection operators satisfying Assumption (A5).

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	related_match	AIO:FederatedLearning covers distributed collaborative model training, whereas PowerGossip is a decentralized peer-to-peer gossip optimization protocol with edge-wise low-rank power-iteration compression.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r556-primary-pdf`	Practical Low-Rank Communication Compression in Decentralized Deep Learning Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi	primary paper full_text	ID: https://dblp.org/rec/conf/nips/VogelsKJ20 sha256:e142a7e3ce9a…
`r556-arxiv-abs`	Practical Low-Rank Communication Compression in Decentralized Deep Learning (arXiv abstract) Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi	primary paper abstract_only	ID: https://arxiv.org/abs/2008.01425 sha256:ce8ea6d08337…
`r556-metadata-cr`	Practical Low-Rank Communication Compression in Decentralized Deep Learning (Crossref metadata) Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi	primary paper abstract_only	ID: https://dblp.org/rec/conf/nips/VogelsKJ20 sha256:8ba4f9db83e4…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T10:39:00Z

Updated: 2026-10-19T10:39:00Z

Source File: data/curated/PowerGossipLowRankCommunicationCompressionForDecentralizedDeepLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories