Quasi-Hyperbolic Momentum (QHM) and Quasi-Hyperbolic Adam (QHAdam) Optimization

Appendix A.5-A.8 and G (Eqs. 5, 7, Fact A.2, p. 16-18, 31) prove that QHM's update direction is a quasi-hyperbolic weighted moving average QHWMA_{nu, beta}(x_{0...t}) = (1 - nu * beta) x_t + nu(1 - beta) sum_{i=1}^t beta^i x_{t-i} with time-inconsistent discount ratios delta(i + tau)/delta(i) = (nu(1 - beta)/(1 - nu * beta)) beta^tau for i = 0 versus beta^tau for i > 0, and that for independent random vectors x_{0...t} with covariance Sigma, lim_{t->infty} Cov[QHWMA_{nu, beta}(x_{0...t})] = rho * Sigma where rho = (1 - nu * beta)^2 + [nu * beta(1 - beta)]^2 / (1 - beta^2), which recovers the EWMA limit covariance (1 - beta)/(1 + beta) at nu = 1 (Fact A.1) and satisfies the tight lower bound rho >= (1 - nu)^2 for any nu < 1.

Section 4 (Table 1, p. 3-5) and Appendix C-D (Eqs. 10-11, p. 20-26) prove via coordinate-wise linear operator transition matrices that QHM (with 1 auxiliary buffer and 3 scaled vector additions) unifies plain SGD (nu = 0), Polyak momentum (nu = 1), and NAG (nu = beta); is bijective with Synthesized Nesterov Variants (SNV, Lessard et al. 2016) via gamma = alpha(1 - beta), beta_1 = beta, beta_2 = (1 - nu)beta/(1 - beta) (thereby recovering Robust Momentum and Triple Momentum with global linear convergence rate 1 - 1/sqrt(kappa) on strongly convex smooth functions); recovers PID control optimizers (Recht 2018; An et al. 2018) with k_P = -alpha * nu * beta/(1 - beta), k_I = alpha, k_D = alpha * nu * beta^2/(1 - beta)^2; and recovers AccSGD (Kidambi et al. 2018) via alpha = delta * epsilon(1 + xi)/(1 + epsilon), nu = (epsilon * xi - 1)/(epsilon(1 + xi)), beta = (kappa - epsilon^2 xi)/(kappa + epsilon * xi) while proving that AccSGD cannot recover NAG because nu = beta requires xi = (1 - epsilon + sqrt(4 kappa + (1 - epsilon)^2))/(2 epsilon) > sqrt(kappa).

Fact F.1 (Appendix F, Eqs. 12-19, p. 28-30) proves that for 0 < beta_1 < sqrt(beta_2) < 1 (with epsilon = 0 and without bias correction), QHAdam's per-coordinate update magnitude is tightly bounded above by ||g_tilde_{t+1} / sqrt(s_tilde_{t+1})||_infty <= sqrt((1 - nu_1 beta_1)^2 / (1 - nu_2 beta_2) + [nu_1 beta_1(1 - beta_1)]^2 [1 - (beta_1^2 / beta_2)^t] / (nu_2(1 - beta_2)(beta_2 - beta_1^2))), attained by the worst-case gradient sequence x_t = (1 - nu_1 beta_1)/(1 - nu_2 beta_2) and x_{t-i} = (nu_1(1 - beta_1)beta_1^i)/(nu_2(1 - beta_2)beta_2^i) (Eq. 17); in the limit t -> infty for vanilla Adam (nu_1 = nu_2 = 1), this yields (1 - beta_1)sqrt(beta_2 / ((1 - beta_2)(beta_2 - beta_1^2))) approx 7.27 alpha at beta_1 = 0.9, beta_2 = 0.999, disproving Kingma & Ba (2015)'s claimed bound alpha * max{1, (1 - beta_1)/sqrt(1 - beta_2)} approx 3.16 alpha and showing that setting nu_2 < 1 in QHAdam tightens the maximum per-step update to prevent gradient-spike explosions.

Across parameter sweeps and large-scale case studies (Section 6, Tables 2-3 & Figures 1-2, p. 6-8; Appendix H, Table 4, p. 32-33), QHM with default nu = 0.7, beta = 0.999 outperforms optimal NAG on EMNIST, CIFAR-10 (PreActResNet18), and ImageNet (ResNet50/ResNet152: 0.2128 vs. 0.2137 top-1 error), improves WikiText-103 FConvLM validation perplexity from 34.92 to 34.45 (at nu = 0.98, beta = 0.998), and outperforms AggMo-Best on EMNIST autoencoders (test MSE 2.0851 vs. 2.2827), while QHAdam improves TD3 MuJoCo continuous control returns and eliminates 4/10 seed explosions on WMT16 EN-DE Transformer translation to reach 29.45 BLEU (vs. 29.17 for Adam).