Home / Curated / Algorithms / Lion (EvoLved Sign Momentum)

Lion (EvoLved Sign Momentum)

Algorithms AIMech:LionEvolvedSignMomentumOptimization DRAFT

Aliases: Lion Optimizer, EvoLved Sign Momentum, Lion EvoLved Sign Momentum Optimizer

A first-order stochastic optimization algorithm (EvoLved Sign Momentum) discovered via evolutionary symbolic program search with abstract execution and funnel selection that tracks only a single exponential moving average momentum buffer m_t and updates parameters using the elementwise sign of a linear interpolation c_t = beta_1 m_{t-1} + (1 - beta_1) g_t combined with decoupled weight decay, theta_t = theta_{t-1} - eta_t (sign(c_t) + lambda theta_{t-1}), followed by a distinct momentum EMA update m_t = beta_2 m_{t-1} + (1 - beta_2) g_t with default beta_1 = 0.9 and beta_2 = 0.99.

Identity Notes & Concept Scope Distinct from Adam, AdamW, AdaBelief, Sophia, and Adafactor in that Lion stores no second-moment or Hessian buffer and applies an elementwise binary sign operation sign(c_t) in {-1, 0, +1} yielding uniform coordinate update magnitudes. Distinct from signSGD with momentum in that Lion decouples the interpolation weight beta_1 = 0.9 used to form the sign-update direction c_t from the longer-horizon EMA decay beta_2 = 0.99 used to update the stored momentum m_t.

Claims & Grounded Evidence

4 Claims

Mechanism #c-r587-dual-interpolation-sign-update

Lion decouples the update direction from momentum tracking via two distinct linear interpolations: it computes c_t = beta_1 m_{t-1} + (1 - beta_1) g_t (default beta_1 = 0.9) before applying the elementwise sign operator theta_t = theta_{t-1} - eta_t (sign(c_t) + lambda theta_{t-1}), and then updates the momentum state with a longer memory horizon m_t = beta_2 m_{t-1} + (1 - beta_2) g_t (default beta_2 = 0.99), outperforming single-EMA sign-momentum ablations with beta_1 = beta_2.

supports Source: Symbolic Discovery of Optimization Algorithms · Locator: Program 1, p. 2; Section 3.1-3.2, p. 7; Section 4.6, p. 13; Algorithm 2, p. 24

Mechanism #c-r587-lr-and-weight-decay-rescaling

Because the elementwise sign update sign(c_t) in {-1, +1} has a larger norm than adaptive gradient updates, Lion requires a learning rate eta_t that is typically 3x to 10x smaller than AdamW and a decoupled weight decay coefficient lambda that is correspondingly 3x to 10x larger so that the effective weight decay strength eta_t * lambda remains invariant.

supports Source: Symbolic Discovery of Optimization Algorithms · Locator: Section 3.2, p. 7; Section 5, p. 14; Table 12, p. 30

Application #c-r587-memory-and-compute-efficiency

By tracking only the momentum tensor m_t and omitting second-moment state v_t, Lion halves the optimizer state memory footprint compared to AdamW and achieves up to 2% higher ViT accuracy on ImageNet, up to 5x pre-training compute savings on JFT, and up to 2.3x training compute reduction with lower FID on diffusion models.

supports Source: Symbolic Discovery of Optimization Algorithms · Locator: Abstract, p. 1; Section 3.2, p. 7; Section 4.1-4.3, pp. 8-11

Limitation #c-r587-small-batch-and-massive-corpus-limitation

Lion's performance advantage over AdamW diminishes when strong data augmentations or small batch sizes (< 64) are used, and its validation perplexity gains are small or not statistically significant on massive high-quality language modeling datasets.

supports Source: Symbolic Discovery of Optimization Algorithms · Locator: Section 1, p. 3; Section 4.6, p. 13; Section 6, pp. 14-15

⚠️ Curated Limitations & Negative Results

Requires rescaling both the learning rate (3-10x smaller) and decoupled weight decay (3-10x larger) relative to AdamW because the sign operation produces unit-magnitude coordinate updates.
Provides smaller or negligible improvements over AdamW when training with small batch sizes (< 64), strong data augmentations, or massive language-modeling corpora.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Lion is a sign-momentum stochastic optimizer for training artificial neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r587-primary-pdf`	Symbolic Discovery of Optimization Algorithms Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V. Le	primary paper full_text	ID: https://dblp.org/rec/conf/nips/ChenLHRW0DLHLL23 sha256:a3a7c83cfe35…
`r587-arxiv-abs`	Symbolic Discovery of Optimization Algorithms (arXiv abstract) Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V. Le	primary paper abstract_only	ID: https://arxiv.org/abs/2302.06675 sha256:45b5a89db71a…
`r587-metadata-cr`	Symbolic Discovery of Optimization Algorithms (Crossref metadata) Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V. Le	primary paper abstract_only	ID: https://dblp.org/rec/conf/nips/ChenLHRW0DLHLL23 sha256:9d6dfba2d24b…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T11:10:00Z

Updated: 2026-10-19T11:10:00Z

Source File: data/curated/LionEvolvedSignMomentumOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories