AdaHessian (Adaptive Second-Order Optimizer)

An adaptive second-order stochastic optimization algorithm with O(d) memory complexity that approximates the Hessian diagonal at step t via Hutchinson's method D_t = z odot (H_t z) using a Rademacher vector z in {-1, +1}^d and a Hessian-free matrix-vector product H_t z = d(g_t^T z)/dtheta (Eqs. 8-9), smooths spatial variance via block-diagonal averaging of block size b, D_t^{(s)}[i b + j] = (1/b) sum_{r=1}^b D_t[i b + r] (Eq. 10), tracks a bias-corrected root-mean-square exponential moving average D_bar_t = sqrt(((1 - beta_2) / (1 - beta_2^t)) sum_{i=1}^t beta_2^{t-i} D_i^{(s)} odot D_i^{(s)}) (Eq. 11), and preconditions the bias-corrected gradient EMA m_t by v_t = (D_bar_t)^k for Hessian power 0 <= k <= 1 via theta_{t+1} = theta_t - eta_t m_t / v_t (Eq. 12, Algorithm 1).