Temporal ensembling
Algorithms AIMech:TemporalEnsembling DRAFT
A semi-supervised classifier-training procedure that regularizes a current stochastic prediction toward a bias-corrected exponential average of earlier predictions for the same indexed training example, alongside labeled cross-entropy.
Identity Notes & Concept Scope
This is the prediction-ensemble algorithm in Laine and Aila, distinguished from the same paper’s Pi-model. Its dataset-indexed stored targets and epoch-based refresh differ from Mean Teacher’s per-step weight averaging. The name does not imply an ensemble of separately trained classifiers at inference. This record describes a training algorithm, not a trained checkpoint or fixed network architecture. The inspected AIO SemiSupervisedLLM class specifically combines LLM pretraining with supervised fine-tuning; its related synonym self-training is not an exact identity for this method. AIO SelfsupervisedLearning excludes human-annotated labels, Distillation describes large-to-small compression, and RegularizationLayer is a layer: none supplies an equivalent class for this algorithm. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 230 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in both worktree and main checkout; matches concern prior discovery leads and the Pi-model normalization regression. Ten exact AIO CLASS nodes, their scoped synonyms and outgoing edges, plus five neighboring native records were inspected. The corrected Unicode-aware normalizer preserves Π; the previous Pi-model/Model collision was a lexical defect, not semantic identity (review issue #58).
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Maintain zero-initialized N-by-C arrays Z and target z_tilde for N indexed examples and C classes. During epoch t, obtain one stochastic prediction z_i for each example when its minibatch is processed. Minimize labeled cross-entropy divided by the full minibatch size plus w(t)/(C|B|) sum over all i in B of ||z_i−z_tilde_i||², holding the stored targets constant during differentiation. After predictions have been used for that epoch, update each accumulated prediction Z_i←alpha Z_i+(1−alpha)z_i and set the next target to Z_i/(1−alpha^t), with fixed 0≤alpha<1 and t starting at 1. Set the first epoch’s consistency weight to zero and ramp it up. Algorithm 2 displays the target update after the epoch; its caption permits per-example updates within a minibatch after the old target has been used. Predictions gathered within an epoch can come from different current parameter states. The exponential average is over example predictions, not model parameters.
⚠️ Curated Limitations & Negative Results
- Per-example prediction memory scales with the number of indexed examples and classes; targets depend on preserving example identity across visits.
- The stated bias correction assumes zero initialization and a fixed alpha; arbitrary momentum schedules or missing-example update schedules need their own accounting.
- Old predictions can remain incorrect or become stale, and input perturbations must preserve the intended class; no unconditional improvement is implied by averaging.
- The record covers the original classifier procedure, not an online-streaming extension, a claim of doubled speed, or reported benchmark accuracy.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:Classification
Classification
|
related_match | This procedure learns class predictions from labeled and unlabeled examples. The pinned AIO target describes a classification task under SupervisedLearning, not this semi-supervised training algorithm; this is an algorithm-to-task association, not equivalence or an is-a assertion. |
AIO:MachineLearningTask
Machine Learning Task
|
related_match | The algorithm specifies how to train a predictor from data. The AIO target is the general learning task; this association does not identify an algorithm with that task or assert an ontology subclass edge. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Samuli Laine, Timo Aila
|
primary paper
full_text
|
sha256:d19ea2ecc468…
|
additional-1
|
Antti Tarvainen, Harri Valpola
|
primary paper
full_text
|
sha256:86c2ad510ea2…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T10:12:04Z
Updated: 2026-10-11T10:12:04Z
Source File:
data/curated/TemporalEnsembling.yaml