Schedule-Free Optimization (Schedule-Free SGD and Schedule-Free AdamW)

A family of anytime stochastic optimization algorithms (Schedule-Free SGD, Equations 3-5, and Schedule-Free AdamW, Algorithm 1) and an online-to-batch conversion framework (Theorem 2 and Theorem 5) that eliminate preset learning rate decay schedules and knowledge of the stopping horizon T without adding extra hyperparameters or memory buffers over standard momentum optimizers. At step t, Schedule-Free optimization forms the gradient evaluation location y_t = (1 - beta) z_t + beta x_t as a convex interpolation between the base optimizer sequence z_t and the online weighted evaluation average x_t, evaluates the stochastic subgradient g_t in partial f(y_t, zeta_t) at y_t, updates the base sequence z_{t+1} (e.g., z_{t+1} = z_t - gamma g_t for SGD or z_{t+1} = z_t - gamma_t g_t / (sqrt{v_t} + epsilon) - gamma_t lambda y_t for AdamW), and updates the evaluation iterate via x_{t+1} = (1 - c_{t+1}) x_t + c_{t+1} z_{t+1} where c_{t+1} = w_{t+1} / sum_{i=1}^{t+1} w_i (such as c_{t+1} = 1/(t+1) or warmup-squared weights c_{t+1} = gamma_t^2 / sum_{i=1}^t gamma_i^2).