Adafactor
Algorithms AIMech:Adafactor DRAFT
An adaptive gradient optimization method that approximates matrix-valued second-moment statistics using row and column factors.
Identity Notes & Concept Scope
Adafactor factors the adaptive second-moment state; it is not low-rank adaptation of model weights such as LoRA. Vector parameters use an unfactored second-moment estimate.
Claims & Grounded Evidence
1 Claim
Mechanism
#update-rule
For matrix parameters, row and column statistics reconstruct a factored second-moment estimate used to normalize gradients; the inspected implementation also clips update RMS and scales steps using parameter RMS.
⚠️ Curated Limitations & Negative Results
- PyTorch documents differences from the original paper in learning-rate control and epsilon handling; the record does not equate their exact update equations.
- No measured memory reduction, benchmark superiority or convergence guarantee is asserted.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:TrainingStrategy
TrainingStrategy
|
related_match | This algorithm can implement parameter fitting, but the pinned TrainingStrategy definition uses a preprocessing genus. This is a contextual training relationship, not equivalence or a subclass assertion. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
pytorch-docs
|
PyTorch contributors
|
official documentation
documentation
|
sha256:1846d2084469…
|
original-paper
|
Noam Shazeer, Mitchell Stern
|
primary paper
full_text
|
ID: arXiv:1804.04235v1
sha256:893ccde9ab41…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T03:25:18Z
Updated: 2026-10-11T03:33:00Z
Source File:
data/curated/Adafactor.yaml