Improved consistency training
Algorithms AIMech:ImprovedConsistencyTraining DRAFT
Aliases: iCT (consistency training)
The consistency-model training procedure of Song and Dhariwal that learns directly from paired shared-noise data perturbations using a stop-gradient teacher with current student parameters, a Pseudo-Huber discrepancy, revised noise weighting and sampling, and a staged increase in noise-discretization intervals.
Identity Notes & Concept Scope
The identity search included 442 AIO records, 90 preceding native identities, all scoped synonyms and aliases, and ignored-inclusive text in both checkouts. This is the named improved consistency-training procedure, not the consistency-model family, the original EMA-teacher training procedure, diffusion-model distillation, or a particular iCT-deep architecture/checkpoint. The acronym is qualified to avoid claiming an unscoped abbreviation.
Claims & Grounded Evidence
1 Claim
Mechanism
#generative-mechanism
The student and stop-gradient teacher evaluate the same data sample plus the same Gaussian noise scaled to adjacent noise levels, with teacher EMA decay set to zero. The revised objective uses a Pseudo-Huber discrepancy and inverse adjacent-noise-gap weighting. Training samples adjacent intervals according to discretized lognormal masses and doubles the interval count at scheduled milestones up to a cap. Noise-embedding sensitivity and shared dropout randomness are additional prescribed training choices; the separate inference-time EMA remains distinct.
⚠️ Curated Limitations & Negative Results
- Removing teacher EMA retains stop-gradient and does not remove inference-time parameter averaging; it is not a blanket recommendation to eliminate EMA from consistency distillation.
- The paper's counterexample diagnoses a flaw in an earlier asymptotic argument. Its correction and finite-noise curriculum do not establish universal finite-training convergence.
- Noise distribution, loss scale, discretization cap, dropout and embedding settings are choices evaluated in particular image-generation settings, not universal hyperparameters. The interval count doubles; the number of noise grid points is that count plus one.
- The record describes the inspected training procedure and retains DRAFT status; it does not reproduce performance comparisons, certify every proof, or identify all future variants with this version.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:TrainingStrategy
TrainingStrategy
|
related_match | The procedure is directly related to choosing how a model is trained. The pinned TrainingStrategy definition calls its subject a preprocessing, whereas iCT is the actual optimization/training procedure; related_match preserves this scope discrepancy instead of asserting an unqualified subclass relation. Distillation is not an equivalent target because this procedure learns directly from data without a separately pretrained diffusion teacher. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
original-paper
|
Yang Song, Prafulla Dhariwal
|
primary paper
full_text
|
ID: arXiv:2310.14189
sha256:2c77b5ae689b…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T04:42:32Z
Updated: 2026-10-11T04:42:32Z
Source File:
data/curated/ImprovedConsistencyTraining.yaml