Home / Curated / Mathematical Methods / InfoNCE

InfoNCE

Mathematical Methods AIMech:InfoNCE DRAFT

Aliases: InfoNCE loss

A contrastive log-softmax objective that identifies a joint positive among sampled candidates and, under independent marginal-negative sampling, gives an expectation-level lower bound on mutual information.

Identity Notes & Concept Scope InfoNCE is the loss introduced in contrastive predictive coding, separable from the CPC encoder/context/prediction procedure. It does not by itself prescribe cosine similarity, temperature, augmentation pairs, symmetric losses or an autoregressive architecture. Named configurations such as SimCLR NT-Xent and DIM’s finite-masking implementation require their own sampling and numerical qualifications. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 203 preceding native IDs/names/aliases. Ignored-inclusive word-delimited searches of both full checkouts returned 52 worktree and 75 main-checkout lines: existing component discussions and discovery/review references, plus unrelated ignored dependency text in main. Short acronyms were compared in normalized identities. Four nearest maintained mechanisms and eleven exact AIO candidate nodes with direct parent edges were inspected.

Claims & Grounded Evidence

1 Claim

Mechanism #method

Given a context c, draw one positive x_+ jointly with c and N−1 negative candidates independently from the marginal p(x), with N>=2. For real scores s(x,c), define ell=−log[exp(s(x_+,c))/sum_(j=1)^N exp(s(x_j,c))], including the positive in the denominator, and minimize L_NCE=E[ell]. For the ideal sampling scheme, log(N)−L_NCE<=I(X;C); since ell>=0, the bound is at most log(N) and may itself be negative. An expressive optimum scores the conditional-to-marginal density ratio up to a context-dependent multiplicative constant in exp(s). The categorical objective remains definable with other candidate constructions, but its stated mutual-information interpretation then requires a separate argument. Poole et al. provide an independent rigorous derivation of the expected bound beyond the approximate step in the original CPC appendix.

supports Source: Representation Learning with Contrastive Predictive Coding · Locator: Section 2.3, PDF pp. 3–4, Equation (2) and density-ratio discussion; Appendix A.1 p. 13, with its approximation explicitly qualified
supports Source: On Variational Bounds of Mutual Information · Locator: Section 2.3, PDF p. 4, Equations (8)–(10), footnote 2 and log(K) ceiling discussion

⚠️ Curated Limitations & Negative Results

The lower bound holds for an expectation under its specified joint and independent marginal sampling. A realized finite-batch number is not certified to be below true mutual information.
The log(N) ceiling limits estimation of high mutual information at a fixed candidate count. Optimizing a finite, restricted critic need not recover the density ratio or true information.
False semantic negatives, conditional negative proposals and correlated in-batch candidates can change the learned representation and invalidate an unqualified use of the ideal sampling proof. Semantic usefulness is not guaranteed by the loss.
Finite masking values contribute nonzero denominator mass and are not algebraically identical to removing candidates. No numerical implementation or training experiment was run for this loss record.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ContrastiveLearning` Contrastive Learning	related_match	InfoNCE is an objective used for positive-versus-negative discrimination in contrastive representation learning. The AIO target is modeled as a network/learning approach, so this mathematical-method-to-approach association is not an is-a relation or an exact match.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`method-source`	Representation Learning with Contrastive Predictive Coding Aaron van den Oord, Yazhe Li, Oriol Vinyals	primary paper full_text	ID: https://arxiv.org/abs/1807.03748v2 sha256:5ef5e1ab47d2…
`additional-1`	On Variational Bounds of Mutual Information Ben Poole, Sherjil Ozair, Aaron van den Oord, Alex Alemi, George Tucker	primary paper full_text	ID: https://proceedings.mlr.press/v97/poole19a.html sha256:38da74b0bae6…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-11T08:44:37Z

Updated: 2026-10-11T08:50:47Z

Source File: data/curated/InfoNCE.yaml