DoG (Distance over Gradients Parameter-Free Step Size Schedule)

A parameter-free dynamic step-size schedule for stochastic gradient descent (x_{t+1} = x_t - eta_t g_t) that sets the initial step size to eta_0 = r_epsilon / ||g_0|| for a small initial movement parameter r_epsilon and dynamically sets subsequent step sizes to the ratio of the maximum observed distance from initialization r_bar_t = max_{i<=t} ||x_i - x_0|| vee r_epsilon to the root-sum-of-squares of observed stochastic gradient norms sqrt(G_t) = sqrt(sum_{i<=t} ||g_i||^2), i.e., eta_t = r_bar_t / sqrt(sum_{i<=t} ||g_i||^2), along with a tamed variant (T-DoG) that inflates the denominator by polylogarithmic factors to guarantee iterate boundedness r_bar_T <= 3 d_0 with high probability and a per-layer variant (L-DoG).