Home / Curated / Algorithms / Adam-mini (Hessian-Block-Partitioned Learning Rate Optimization)

Adam-mini (Hessian-Block-Partitioned Learning Rate Optimization)

Algorithms AIMech:AdamMiniHessianBlockPartitionedLearningRateOptimization DRAFT

Aliases: Adam-mini, Adam-mini Optimizer, Hessian-Structure Blockwise Adaptive Learning Rate Optimizer, Smallest-Dense-Hessian-Subblock AdamW

A memory-efficient adaptive first-order optimization algorithm that partitions neural network parameters into blocks corresponding to the smallest dense sub-blocks of the Hessian matrix and replaces Adam's coordinate-wise second-moment vector within each block by a single scalar exponential moving average of the block-mean squared gradient, eliminating over 99.9% of second-moment parameters and halving total optimizer-state memory.

Identity Notes & Concept Scope Distinct from NovoGrad (AIMech:NovoGrad), which partitions parameters coarsely by entire PyTorch layers (causing training instability on billion-parameter Transformers) and normalizes the current gradient before accumulating first-order momentum; Adam-mini partitions Transformer parameters by smallest dense Hessian sub-blocks (heads for Query/Key, output neurons for Value/Attn.Proj/MLP, and tokens for Embedding/Output) and retains standard AdamW first-order momentum m_b while averaging g_b (.) g_b per block. Distinct from LAMB (AIMech:LambLayerwiseAdaptiveLargeBatchOptimization), which retains full coordinate-wise v in R^d alongside layerwise trust-ratio scaling and saves zero memory.

Claims & Grounded Evidence

5 Claims

Mechanism #adammini-hessian-subblock-partition-principle-mechanism

Following Principle 1 (partitioning parameters into blocks associated with the smallest dense sub-blocks of the neural network Hessian), Adam-mini partitions Transformer parameters via Algorithm 3 by tokens for embedding and output layers, by attention heads for query and key projections, and by output neurons for value, attention output projection, and MLP layers.

supports Source: Adam-mini: Use Fewer Learning Rates To Gain More · Locator: Section 2.3 Principle 1, Figure 7, and Appendix B Algorithm 3, pages 6-7, 19

Mechanism #adammini-block-mean-second-moment-update-mechanism

For each parameter block b in {1, ..., B} with gradient g_b, Adam-mini applies decoupled weight decay, updates full coordinate-wise first-order momentum m_b = (1 - beta_1) * g_b + beta_1 * m_b with m_hat_b = m_b / (1 - beta_1^t), tracks a single scalar second moment v_b = (1 - beta_2) * mean(g_b * g_b) + beta_2 * v_b with v_hat_b = v_b / (1 - beta_2^t), and updates block parameters via param_b <- param_b - eta_t * m_hat_b / (sqrt(v_hat_b) + epsilon), cutting >=99.9% of Adam's v entries and reducing optimizer-state memory by 50%.

supports Source: Adam-mini: Use Fewer Learning Rates To Gain More · Locator: Section 1 page 2, Section 2.2 Algorithm 1 page 5, and Section 2.4 Table 1 page 7

Scope #adammini-dense-hessian-condition-number-scope

On dense positive-definite Hessian sub-blocks H_b with low diagonal-over-off-diagonal ratio tau = (sum_i |H_{b,i,i}|) / (sum_{i,j} |H_{b,i,j}|), Adam's coordinate-wise diagonal preconditioner D_Adam = Diag(1 / sqrt(v)) fails to reduce and often increases the condition number r = kappa(D_Adam H_b) / kappa(H_b) > 1, whereas a single scalar learning rate per dense sub-block preserves block conditioning and outperforms coordinate-wise Adam.

supports Source: Adam-mini: Use Fewer Learning Rates To Gain More · Locator: Section 2.1 Equation (2), Figures 4-5, pages 3-5, and Appendix D.1 Table 3, page 20

Application #adammini-llm-pretraining-sft-rlhf-and-throughput-application

Across GPT-2 (125M to 1.5B) and LLaMA-2/3 (20M to 13B) pre-training, Chinchilla scaling-law fits, LLaMA-2-7B SFT and ReMax RLHF alignment, and ResNet/Swin/DiT/DDPM/GAT/GCN training, Adam-mini matches or outperforms AdamW using identical hyperparameters while halving optimizer memory and achieving 49.6% higher throughput (33.1% wall-clock GPU-hour reduction) on LLaMA-2-7B.

supports Source: Adam-mini: Use Fewer Learning Rates To Gain More · Locator: Section 2.4 Tables 1-2 page 7, Section 3.1-3.3 Figures 8-12 pages 8-10, and Appendix D.3-D.5 Tables 4-7 pages 21-23

Limitation #adammini-coarse-partition-instability-and-first-moment-memory-limitation

Coarse layer-wise partitioning (such as PyTorch default parameter groups) violates Principle 1 on Transformers and causes severe training loss spikes, and because Adam-mini retains full coordinate-wise first-order momentum m, its optimizer-state memory reduction is capped at 50% unless combined with orthogonal gradient-projection or quantization methods.

supports Source: Adam-mini: Use Fewer Learning Rates To Gain More · Locator: Section 2.3 Figure 7(i) pages 6-7, Section 3.1 Figure 8(a) page 8, and Appendix A page 18

⚠️ Curated Limitations & Negative Results

Requires architecture-aware Hessian sub-block partitioning (Principle 1); naive PyTorch layer-level partitioning causes loss spikes on large Transformers.
Retains full coordinate-wise first-order momentum m, bounding standalone optimizer-state memory savings at 50%.
Approximates each dense Hessian block's step size via the arithmetic mean of squared gradients rather than exact block spectral quantities.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Adam-mini is a memory-efficient adaptive optimizer for pre-training, supervised fine-tuning, and RLHF alignment of Transformer language models, diffusion models, vision backbones, and graph neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r638-primary-pdf`	Adam-mini: Use Fewer Learning Rates To Gain More Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, Ruoyu Sun	primary paper full_text	ID: https://dblp.org/rec/conf/iclr/ZhangCLD0K0L025 sha256:d64fd863d8a7…
`r638-arxiv-abs`	Adam-mini: Use Fewer Learning Rates To Gain More (arXiv abstract) Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, Ruoyu Sun	primary paper abstract_only	ID: https://arxiv.org/abs/2406.16793 sha256:ebb5eb2e3d28…
`r638-metadata-cr`	Adam-mini: Use Fewer Learning Rates To Gain More (Crossref metadata) Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, Ruoyu Sun	primary paper abstract_only	ID: https://dblp.org/rec/conf/iclr/ZhangCLD0K0L025 sha256:54882545dca1…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:01:00Z

Updated: 2026-10-19T12:01:00Z

Source File: data/curated/AdamMiniHessianBlockPartitionedLearningRateOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories