PowerSGD Low-Rank Subspace Iteration Gradient Compression with Error Feedback

A distributed low-rank gradient compression algorithm (PowerSGD) that reshapes each parameter tensor's error-compensated gradient matrix Delta_w = g_{t,w} + e_{t,w} in R^{n x m} on worker w in [W], computes a single warm-started rank-r block power iteration step via P_w = Delta_w Q, all-reduces P = (1/W) sum_{w=1}^W P_w, orthonormalizes the columns of P via Gram-Schmidt, computes Q_w = Delta_w^T P, all-reduces Q = (1/W) sum_{w=1}^W Q_w, reconstructs the rank-r update Delta_hat = P Q^T, and stores the local post-compression residual e_{t+1,w} = Delta_w - P Q_w^T, exploiting the linearity of matrix-subspace projections to aggregate compressed updates via all_reduce in O((n + m)*r) communication with exact single-worker equivalence.