VR-DIANA and SVRG-DIANA (Variance-Reduced Distributed Learning with Gradient Quantization)

VR-DIANA and SVRG-DIANA are doubly variance-reduced distributed optimization algorithms for solving min_{x in R^d} f(x) = (1/n) sum_{i=1}^n f_i(x) with local finite-sum structure f_i(x) = (1/m) sum_{j=1}^m f_{ij}(x) across n workers using arbitrary unbiased omega-quantization operators Q: R^d -> R^d (satisfying E_Q[Q(v)] = v and E_Q[||Q(v) - v||_2^2] <= omega ||v||_2^2). At each iteration k, each worker i computes a local variance-reduced stochastic gradient g_i^k = grad f_{i, j_i^k}(x^k) - grad f_{i, j_i^k}(w_{i, j_i^k}^k) + mu_i^k via L-SVRG (Variant 1), SAGA (Variant 2), or multi-epoch SVRG (Algorithm 3), quantizes only the shifted difference Delta_hat_i^k = Q(g_i^k - h_i^k) relative to a local state h_i^k, updates h_i^{k+1} = h_i^k + alpha * Delta_hat_i^k, and enables the master to take the global step x^{k+1} = x^k - gamma * (h^k + (1/n) sum_{i=1}^n Delta_hat_i^k) that eliminates both local sampling variance and quantization variance without ever transmitting uncompressed gradients.