Gumbel-Softmax relaxation
Mathematical Methods AIMech:GumbelSoftmaxRelaxation DRAFT
Aliases: Gumbel-Softmax, Concrete distribution, Concrete relaxation, Gumbel-Softmax distribution
A continuous relaxation of a finite categorical random variable obtained by applying softmax to independent Gumbel-perturbed log weights divided by a positive temperature, producing a differentiable random vector in the simplex interior.
Identity Notes & Concept Scope
Gumbel-Softmax and Concrete identify the same positive-temperature distributional construction: the two inspected primary papers explicitly acknowledge the concurrent work, and their sampling formulas agree after normalizing positive weights. This record covers that shared relaxation and its pathwise-gradient use. It does not equate all training objectives in the papers, the hard Gumbel-Max sample, generic deterministic softmax, or the separate straight-through estimator. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 331 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in worktree and main, returning 11 and 11 lines respectively. All 10 unique hit contents were inspected: neighboring Gumbel-Top-k contrasts, saved review and ledger references, new source leads, and incidental concrete-distribution wording in an older random-search report, not existing candidate records. Exact checkout counts and differences are retained in the ledger. Nine exact AIO CLASS nodes with scoped synonyms/comments and outgoing edges and six full native neighboring records were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
3 Claims
Mechanism
#method
For n≥2 categories with positive finite weights alpha_i and temperature tau>0, draw independent standard Gumbel G_i and set y_i=exp((log(alpha_i)+G_i)/tau) divided by the sum of those exponentials. Normalizing alpha beforehand only adds a common log shift and leaves y unchanged. Each draw has positive components summing to one and is differentiable in the log weights at positive temperature. Holding the base noise fixed allows the chain rule to estimate gradients of a differentiable objective evaluated on this relaxed random vector, under the usual conditions permitting differentiation under expectation. Argmax rounding has categorical probabilities alpha_i/sum(alpha) for every positive temperature; using the same noise while tau tends to zero yields the corresponding one-hot sample almost surely. For fixed finite perturbed scores, tau tending to infinity yields the uniform vector. The relaxed vector at finite temperature is not itself a categorical one-hot draw.
Scope
#distribution-identity
The Gumbel-Softmax paper explicitly calls the independently discovered distribution Concrete. Its normalized-probability formula and the Concrete paper’s unnormalized-weight formula produce the same simplex random vector after the common normalization shift cancels. The Concrete discussion distinguishes this shared distribution from choices of relaxed training objective.
Limitation
#objective-boundary
For variational training, the Concrete paper uses densities matching the relaxed prior and posterior in its relaxed objective, then evaluates the discrete objective at test time. It warns that naive insertion of soft states into discrete log-mass terms need not preserve the intended lower-bound interpretation. This source-specific distinction is not an assertion that every version of every Gumbel-Softmax implementation uses the same loss.
⚠️ Curated Limitations & Negative Results
- Pathwise gradients can be unbiased for the specified continuous relaxed objective under regularity conditions while remaining biased for the original discrete objective. A correct categorical argmax law does not make the relaxed gradient an unbiased discrete-gradient estimator.
- Temperature zero is a limit, not a valid value in the softmax division. Small temperatures can produce sharp derivatives and difficult variance or numerical behavior; no universally optimal temperature or annealing schedule is asserted.
- The positive-weight construction excludes zero-mass categories or requires a declared limiting representation. Finite-precision softmax, noise endpoints and ties require numerical policies.
- A uniform vector at very high temperature is different from a uniform probability density across the simplex. The relaxation is not generally a Dirichlet distribution.
- Distributional equivalence does not imply equivalence of training losses. When a variational objective includes latent log probabilities, replacing discrete states by soft values without a consistent density treatment need not preserve a lower bound for the intended discrete model.
- A downstream objective must be defined on the continuous simplex for relaxed training. A model may exploit interior values and suffer a gap when later evaluated with discrete states; neither task quality nor unbiased discrete optimization is guaranteed.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:SoftmaxFunction
Softmax Function
|
related_match | The inspected construction explicitly uses softmax on temperature-scaled Gumbel-perturbed log weights, and differentiates that smooth transformation. The exact pinned AIO node denotes the normalization function. This is a method-to-essential-component related match, not equivalence, subclass membership or a claim that the stochastic construction is merely ordinary deterministic softmax. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Eric Jang, Shixiang Gu, Ben Poole
|
primary paper
full_text
|
ID: arXiv:1611.01144v5
sha256:e424dbd5b4e8…
|
concrete-source
|
Chris J. Maddison, Andriy Mnih, Yee Whye Teh
|
primary paper
full_text
|
ID: arXiv:1611.00712v3
sha256:56864ffef78f…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T16:30:45Z
Updated: 2026-10-11T16:30:45Z
Source File:
data/curated/GumbelSoftmaxRelaxation.yaml