Cautious Optimizers (C-AdamW / C-Lion Alignment-Masked Momentum)

In continuous-time Hamiltonian+Descent systems d/dt w_t = -grad K(s_t) - Phi_t(grad L(w_t)), d/dt s_t = grad L(w_t) - Psi_t(grad K(s_t)) with Hamiltonian H(w, s) = L(w) + K(s), Theorem 2.1 proves that weighting grad K(s_t) by phi(x_t) where x_t = grad L(w_t) circ grad K(s_t) and phi(x)_i >= 1 for x_i > 0, phi(x)_i <= 0 for x_i < 0 guarantees both d/dt H <= -Delta_H <= 0 and monotonic loss decrease d/dt L <= -x_t^T phi(x_t) - ||grad L||_{Phi_t}^2 <= 0 while converging to stationary points of H(w, s) (Corollary 2.2, pp. 4-5, 14-16).

For a mu-smooth loss function L, Theorem 2.3 establishes that whenever Delta(x) := -x^T (1 - phi(x)) >= 0 and the step size satisfies epsilon_t <= 2 Delta(u_t circ g_t) / (mu ||r_t|| (2 ||u_t|| + ||r_t||)) with r_t = u_t circ (1 - phi_t), a single cautious step achieves strictly lower or equal objective loss than the unmasked base optimizer step, L(w_bar_{t+1}) <= L(w_{t+1}) (Section 2.3 & Appendix A.4, pp. 5, 17-18).

C-AdamW and C-Lion consistently improve evaluation perplexity over AdamW and Lion on LLaMA pretraining across C4 (60M-100M) and FineWeb-Edu (130M-1.2B, improving downstream accuracy on 5 of 7 LM Eval-Harness benchmarks), improve ViT Top-1 accuracy on Mini-ImageNet across C-AdamW (73.52% vs 72.11%), C-LaProp (73.92% vs 71.73%), and C-MARS (74.91% vs 74.06%), and accelerate MAE pretraining and Qwen2.5/Pythia post-training (Section 3.2-3.3 & Appendix E, pp. 7-8, 24-25).

Discrete-time dominance over the base optimizer (Theorem 2.3) is guaranteed per step within a bounded step-size range rather than globally across multi-step trajectories on arbitrary nonconvex landscapes, and computing the global active-mask mean ||phi_t||_0 / d incurs an extra element-wise mask pass (~3% throughput reduction vs fused PyTorch AdamW) and a scalar all-gather under tensor parallelism.