CAME (Confidence-guided Adaptive Memory Efficient Optimization)
Algorithms AIMech:CAMEConfidenceGuidedAdaptiveMemoryEfficientOptimization DRAFT
Aliases: CAME Optimizer, Confidence-guided Adaptive Memory-Efficient Optimizer, Confidence-Guided Adafactor (CAME)
A memory-efficient adaptive optimization algorithm (Algorithm 2) that extends Adafactor by computing the elementwise squared residual instability matrix U_t = (u_hat_t - m_t)^2 between the RMS-clipped rank-1 second-moment update u_hat_t and its exponential moving average m_t = beta_1 m_{t-1} + (1 - beta_1) u_hat_t, factorizing U_t into row and column moving-average vectors R_t in R^n and C_t in R^m via generalized Kullback-Leibler rank-1 non-negative matrix factorization S_t = R_t C_t / (1_n^T R_t), and scaling the momentum step by the confidence factor 1 / sqrt(S_t) via theta_t = theta_{t-1} - (eta / sqrt(S_t)) m_t.
Identity Notes & Concept Scope
Distinct from Adafactor (Algorithm 1) in that Adafactor directly applies theta_t = theta_{t-1} - eta m_t without accounting for the deviation between m_t and u_hat_t caused by rank-1 second-moment NMF errors, whereas CAME divides m_t by sqrt(S_t) where S_t is a rank-1 NMF reconstruction of the instability matrix U_t = (u_hat_t - m_t)^2. Distinct from Adam and AdaBelief in that both the second moment v_t and the instability matrix S_t are stored as rank-1 vector factors (r_t, c_t, R_t, C_t) in O(n + m) auxiliary second-order memory rather than O(n m) full matrices.
Claims & Grounded Evidence
3 Claims
Mechanism
#c-r598-confidence-guided-instability-nmf-scaling
By measuring the squared deviation U_t = (u_hat_t - m_t)^2 between the current clipped Adafactor update u_hat_t and its EMA m_t and factorizing U_t into rank-1 moving averages R_t = beta_3 R_{t-1} + (1 - beta_3)(U_t + epsilon_2 1_n 1_m^T) 1_m and C_t = beta_3 C_{t-1} + (1 - beta_3) 1_n^T (U_t + epsilon_2 1_n 1_m^T) with S_t = R_t C_t / (1_n^T R_t) (Algorithm 2), CAME suppresses steps when update error is large (low confidence 1 / sqrt(S_t)) and amplifies steps when u_hat_t closely matches m_t, requiring only O(n + m) additional memory over Adafactor.
Application
#c-r598-bert-gpt2-t5-convergence-and-memory-reduction
CAME reduces optimizer state memory by 47% on BERT-4B (from 30.6 GB for Adam to 16.3 GB, within 1% of Adafactor's 15.9 GB; Figure 7 & Table 1), improves BERT-Large masked LM validation accuracy over Adafactor from 63.1% to 66.5% at batch size 8K and from 61.9% to 68.0% at batch size 32K (improving GLUE/SQuAD average by +0.7/+0.5 points, Table 2), and matches Adam validation perplexity on GPT-2 (50.1 vs Adafactor's 56.9) and T5-Base.
Limitation
#c-r598-extra-nmf-cost-and-missing-formal-proof-limitation
CAME incurs additional per-step computation for the second rank-1 non-negative matrix factorization over U_t compared with Adafactor, exhibits minor accuracy degradation on 32K-batch BERT-Large relative to full-matrix LAMB, and lacks a formal theoretical convergence rate proof.
⚠️ Curated Limitations & Negative Results
- Still requires storing the first-moment buffer m_t of size O(n m), so total optimizer memory is O(n m + 2(n + m)) rather than sublinear.
- Incurs extra elementwise and row/column reduction operations to factorize the instability matrix U_t at every step and lacks formal convergence rate guarantees (Section 6, p. 9).
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | CAME is a confidence-guided memory-efficient adaptive optimizer for pre-training and fine-tuning large language models and deep artificial neural networks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r598-primary-pdf
|
Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, Yang You
|
primary paper
full_text
|
ID: https://dblp.org/rec/conf/acl/LuoRZJ0023
sha256:342fee8af63f…
|
r598-arxiv-abs
|
Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, Yang You
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2307.02047
sha256:58a597197a66…
|
r598-metadata-cr
|
Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, Yang You
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/conf/acl/LuoRZJ0023
sha256:fdbd9433e995…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T11:21:00Z
Updated: 2026-10-19T11:21:00Z
Source File:
data/curated/CAMEConfidenceGuidedAdaptiveMemoryEfficientOptimization.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories