GaLore (Gradient Low-Rank Projection)

A memory-efficient full-parameter training strategy (Definitions 3.6 and Algorithm 2) for large language models and deep neural networks that projects each layer's full-rank weight gradient G_t = -nabla_W phi_t(W_t) in R^{m x n} (with m <= n) into a compact low-rank representation R_t = P_t^T G_t Q_t in R^{r x r} (or one-sided R_t = P_t^T G_t in R^{r x n} with r << m) using semi-orthogonal projection matrices P_t in R^{m x r}, Q_t in R^{n x r} obtained from the top-r singular vectors of G_t and refreshed every T iterations. GaLore tracks entry-wise stateful optimizer statistics (such as Adam's first and second moments M_t, V_t in R^{r x n}) exclusively on the projected gradient core R_t to form the normalized low-rank update N_t = rho_t(R_t) = hat{M}_t / (sqrt{hat{V}_t} + epsilon) and projects back to the original parameter space via tilde{G}_t = alpha P_t N_t Q_t^T (or alpha P_t N_t) to update the full-rank weight matrix W_t = W_{t-1} + eta tilde{G}_t, traversing a sequence of low-rank subspaces W_t = W_0 + Delta W_{T_1} + ... + Delta W_{T_k} to achieve full-rank learning with O(mn + mr + 2nr) memory.