Whitening mean squared error
Algorithms AIMech:WhiteningMSE DRAFT
Aliases: W-MSE, Whitening MSE
A self-supervised representation-learning procedure that differentiably whitens projected image-view embeddings and minimizes the average squared distance between normalized whitened embeddings of views from the same image.
Identity Notes & Concept Scope
W-MSE refers to Ermolov et al.’s normalized whitening mean-squared-error method. It is not generic MSE, a standalone whitening transform, Barlow Twins cross-correlation matching or VICReg soft covariance regularization. The unnormalized Euclidean ablation is distinguished from the normalized default. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 195 prior native IDs/names/aliases. Ignored-inclusive, word-delimited searches of both full checkouts found nine prior discovery/review references in each plus six unrelated language-stemmer tokens in the main environment. An initial substring search also matched unrelated layer-name suffixes. Neighboring k-means, SwAV, Barlow Twins and VICReg mechanisms and exact AIO definitions/edges were inspected; shared components do not establish identity.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Generate d≥2 augmented views of each of N images and pass them through a shared encoder and projection head. For a whitening group of m projected vectors in R^p, center by the group mean and compute sample covariance Sigma=sum_i(v_i−mean)(v_i−mean)^T/(m−1). When Sigma is positive definite, factor Sigma=L L^T and transform each vector to z_i=L^−1(v_i−mean). For nonzero z normalize u_i=z_i/||z_i|| and minimize the average ||u_i−u_j||^2 over all N d(d−1)/2 unordered same-image view pairs. Backpropagate through both views and whitening, including its batch statistics. In batch slicing, place each image’s views in separate view partitions, apply the same random image-index permutation to every partition, split each into groups, and whiten each group separately; compare corresponding positive views afterward. Repeat slicing and average losses if configured. The procedure requires no negative-pair loss, prediction head, momentum teacher or stopped target.
⚠️ Curated Limitations & Negative Results
- Exact Cholesky whitening requires positive-definite covariance. A centered group has rank at most m−1, so m>p is necessary but not sufficient. The paper’s group-size heuristic is not a rank or numerical-stability guarantee.
- The pinned code optionally replaces Sigma by (1−epsilon)Sigma+epsilon I. Whitening that regularized matrix does not generally give the original centered samples identity covariance. A collapsed group is outside the exact unregularized construction; numerical regularization does not prove every optimizer avoids collapse.
- Identity covariance expresses second-order decorrelation and unit variances. It does not prove statistical independence, Gaussianity or a rotationally symmetric distribution. Subsequent L2 normalization generally changes covariance again.
- Whitening is group-dependent: absence of explicit negatives does not mean independent per-example training. Group composition, common view permutations, embedding dimension and repeated slicing affect the computation. The pinned slicing implementation requires the image batch size to be divisible by its group size.
- Zero-norm handling and covariance regularization matter. The main record states normalized squared Euclidean distance; the supplement’s unnormalized variant and any element-averaged MSE convention are not silently identified. No runtime, gradient implementation or benchmark was reproduced.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:SelfsupervisedLearning
Self-supervised Learning
|
related_match | This algorithm learns from self-generated targets or relationships between unlabeled inputs. The pinned AIO target is a MachineLearningTask; this is a related algorithm-to-task mapping, not equivalence or an is-a assertion. |
Evidence Sources & Primary Literature
5 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe
|
primary paper
full_text
|
ID: https://proceedings.mlr.press/v139/ermolov21a.html
sha256:cef9a776e57d…
|
additional-1
|
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe
|
primary paper
full_text
|
ID: https://proceedings.mlr.press/v139/ermolov21a/ermolov21a-supp.pdf
sha256:f34fa953f14a…
|
additional-2
|
htdt/self-supervised contributors
|
official documentation
documentation
|
ID: https://github.com/htdt/self-supervised/blob/d9662c8d07dafd194a9045375f4f6aa09f5b03e9/methods/w_mse.py
sha256:70f50c710e1a…
|
additional-3
|
htdt/self-supervised contributors
|
official documentation
documentation
|
ID: https://github.com/htdt/self-supervised/blob/d9662c8d07dafd194a9045375f4f6aa09f5b03e9/methods/whitening.py
sha256:d438ac9a9d5e…
|
additional-4
|
htdt/self-supervised contributors
|
official documentation
documentation
|
ID: https://github.com/htdt/self-supervised/blob/d9662c8d07dafd194a9045375f4f6aa09f5b03e9/methods/norm_mse.py
sha256:9ad056410ae5…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T08:15:29Z
Updated: 2026-10-11T08:15:29Z
Source File:
data/curated/WhiteningMSE.yaml