Mime Framework for Mimicking Centralized Stochastic Algorithms in Federated Learning

A general federated optimization framework (Mime and MimeLite) that adapts arbitrary centralized stateful stochastic optimizers A = (U, V) (including Heavy-Ball momentum, Adam, Adagrad, and momentum-based variance reduction MVR) to cross-device federated learning by decomposing A into a linear parameter step U(g, s) and a statistics update V(g, s), updating the optimizer statistics s_t = V(c_t, s_{t-1}) only once per communication round at the server using the unbiased cross-client average gradient c_t = (1/|S_t|) sum_{i in S_t} nabla f_i(x_{t-1}), broadcasting (x_{t-1}, s_{t-1}, c_t) so each sampled client i in S_t executes K local parameter steps y_{i,k} = y_{i,k-1} - eta * U(g_{i,k}, s_{t-1}) with frozen server statistics s_{t-1} and either SVRG variance-reduced local gradients g_{i,k} = nabla f_i(y_{i,k-1}; zeta_{i,k}) - nabla f_i(x_{t-1}; zeta_{i,k}) + c_t (Mime) or raw local stochastic gradients g_{i,k} = nabla f_i(y_{i,k-1}; zeta_{i,k}) (MimeLite).