Deep Gradient Compression (DGC with Momentum Correction and Momentum Factor Masking)

A high-sparsity distributed gradient compression algorithm (Lin et al. 2018, Section 3, Algorithm 1) that transmits only the top-k magnitude entries (down to 0.1% density) of a locally accumulated velocity buffer while preserving dense momentum-SGD convergence via four coupled mechanisms: (1) Momentum Correction (u_{k,t} = m u_{k,t-1} + g_{k,t}, v_{k,t} = v_{k,t-1} + u_{k,t}, Eq. 6), which accumulates local momentum velocities rather than raw gradients; (2) Momentum Factor Masking (v_{k,t} <- v_{k,t} o (1 - Mask), u_{k,t} <- u_{k,t} o (1 - Mask), Eq. 8), which clears both the accumulator and momentum factor on transmitted coordinates; (3) local gradient clipping scaled by N^{-1/2} before accumulation; and (4) warm-up sparsity scheduling.