Home / Curated / Algorithms / VICReg

VICReg

Algorithms AIMech:VICReg DRAFT

Aliases: Variance-invariance-covariance regularization

A self-supervised joint-embedding procedure that combines a paired-view invariance loss with separate per-branch penalties for insufficient coordinate standard deviation and off-diagonal covariance.

Identity Notes & Concept Scope This is VICReg as defined by Bardes et al., not a generic covariance penalty, Barlow Twins cross-correlation loss, VICRegL or a checkpoint. Both branches are optimized through their outputs; shared weights and identical backbones are supported but are not requirements of the general construction. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 191 prior native IDs/names/aliases. Ignored-inclusive searches of both full checkouts found only prior discovery leads (four hits per checkout) and one unrelated main-environment DFSWAVE token. Neighboring SimCLR, MoCo, BYOL and DINO procedures and exact AIO definitions/edges were inspected; shared components do not establish identity.

Claims & Grounded Evidence

1 Claim

Mechanism #method

Encode paired augmented views and pass their representations through expander networks to obtain n-by-d embeddings Z and Z′, n≥2. In the paper’s Equations (1)–(6), define s(Z,Z′)=sum_i||z_i−z′_i||^2/n; v(Z)=sum_j max(0,gamma−sqrt(Var(Z_:j)+epsilon))/d with epsilon>0 and a target gamma>0; and c(Z)=sum_(j≠k) C(Z)_jk^2/d, where C(Z)=sum_i(z_i−mean(Z))(z_i−mean(Z))^T/(n−1). Use sample variance consistent with the covariance diagonal and the supplement’s variance call. Minimize lambda s+mu[v(Z)+v(Z′)]+nu[c(Z)+c(Z′)] for positive weights, differentiating through both branches without target stop-gradient. Branches must produce matching embedding dimensions but can have different encoder architectures or weights. The loss requires no output normalization, negative-pair dictionary or momentum teacher. After pretraining discard the expanders and retain encoder representations.

supports Source: VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning · Locator: Sections 4–4.2, Equations (1)–(6), PDF pp. 4–5; Section 5.3 multimodal scope, p. 7; Appendix A Algorithm 1, p. 13
context Source: VICReg author training implementation · Locator: main_vicreg.py lines 183–222 (forward loss, feature gathering, half-summed variance penalties and covariance scaling) and 307–323 (FullGatherLayer forward/backward)
context Source: PyTorch functional mean squared error API semantics · Locator: Function signature and reduction argument: default mean of elementwise squared error
context Source: PyTorch variance API semantics · Locator: Function signature, sample-variance formula, correction=1 default and pre-2.0 unbiased argument equivalence

⚠️ Curated Limitations & Negative Results

The pinned author code uses element-averaged F.mse_loss and averages the two variance penalties with an additional factor 1/2. The main equations instead use a per-image squared Euclidean sum and a sum of branch variance penalties; coefficient values are not interchangeable without corresponding rescaling.
A hinge on regularized standard deviation stops penalizing values above its threshold; it does not enforce equality to gamma or a hard variance constraint. A positive penalty at zero sample variance requires gamma>sqrt(epsilon), as in the reported gamma=1 and epsilon=0.0001 setting. A positive penalty at a collapsed output does not prove that every optimizer can escape it.
The loss does not require batch normalization, but the reported expander architecture uses it internally. Absence of output normalization is not a claim that every layer is normalization-free.
Covariance decorrelation does not establish statistical independence. With centered n-sample embeddings, rank limits can make simultaneous positive variance in all d coordinates and exactly zero off-diagonal covariance unattainable when d>n−1.
The pinned distributed implementation gathers embeddings with backward support and uses the configured global batch size in the covariance denominator. Framework defaults, batch composition, augmentation, dimensions and coefficients matter to reproduction. No historical runtime execution, general noncollapse guarantee or numerical benchmark is asserted.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:SelfsupervisedLearning` Self-supervised Learning	related_match	This is an algorithm for learning representations from relationships between unlabeled views. The pinned AIO target is a MachineLearningTask; this is a related algorithm-to-task mapping, not equivalence or an is-a assertion.

Evidence Sources & Primary Literature

4 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`method-source`	VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning Adrien Bardes, Jean Ponce, Yann LeCun	primary paper full_text	ID: https://arxiv.org/abs/2105.04906v3 sha256:c227c290d5eb…
`additional-1`	VICReg author training implementation Facebook Research vicreg contributors	official documentation documentation	ID: https://github.com/facebookresearch/vicreg/tree/4e12602fd495af83efd1631fbe82523e6db092e0 sha256:88c4439e2c50…
`additional-2`	PyTorch functional mean squared error API semantics PyTorch contributors	official documentation documentation	ID: https://docs.pytorch.org/docs/2.14/generated/torch.nn.functional.mse_loss.html
`additional-3`	PyTorch variance API semantics PyTorch contributors	official documentation documentation	ID: https://docs.pytorch.org/docs/2.14/generated/torch.var.html

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-11T08:04:24Z

Updated: 2026-10-11T08:05:31Z

Source File: data/curated/VICReg.yaml