Stochastic-times-smooth unit
Mathematical Methods AIMech:StochasticTimesSmoothUnit DRAFT
Aliases: STS unit
A stochastic activation that sets p=sigmoid(a), samples a Bernoulli gate with success probability sqrt(p), and outputs the gate multiplied by sqrt(p).
Identity Notes & Concept Scope
This record covers Bengio et al.’s Section 2.2 STS construction and its smooth-amplitude training path. The Bernoulli gate and nonzero amplitude are both sqrt(p); it is neither an ordinary Bernoulli(p) output nor fixed-rate dropout. The source’s Figure 2 uses the reversed wording Smooth Times Stoch. for this experimental unit family. The identity is not the generic straight-through estimator or stochastic depth. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 337 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in worktree and main, returning 22 and 22 lines respectively. All 15 unique matched line contents were inspected in full: the preceding gradient-estimator records, their cited Bengio source, saved source-reading scopes, reviews and open discovery leads, not existing candidate records. Exact checkout counts and differences are retained in the ledger. Thirteen exact AIO CLASS nodes with scoped synonyms/comments and outgoing edges and six full native neighboring records were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Compute p=sigmoid(a), draw b in {0,1} with P(b=1)=sqrt(p), and return h=b*sqrt(p). Conditional on a, E[h]=p, P(h>0)=sqrt(p), and the nonzero output is sqrt(p). Backpropagation through the smooth multiplier can hold the discrete gate fixed. This path does not include the derivative of the gate’s probability law: as a curator consequence of the displayed rule, its expected derivative for the loss h is p-prime/2, while d E[h]/d a=p-prime. Proposition 2 gives an asymptotic relationship E[f(h)]=f(p)+o(sqrt(p)) as p tends to zero for the stated differentiable-function setting; it does not state equality at a finite activation probability.
⚠️ Curated Limitations & Negative Results
- The smooth-amplitude backward path is not generally an unbiased gradient of the expected stochastic loss. An estimator for changes in the gate probability would require an additional treatment.
- Matching the mean to p does not preserve a nonlinear downstream function. For example, at p=1/4 the second moment is 1/8 while the squared mean is 1/16; this is a curator calculation from the source’s distribution.
- The asymptotic small-p statement is not a uniform finite-p error bound, a guarantee about optimization, or a theorem that the stochastic network equals a deterministic sigmoid network.
- The conditional-computation motivation does not establish a universal speed or accuracy improvement. No author implementation or trained model was run.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:SigmoidFunction
Sigmoid Function
|
related_match | The specified construction uses the pinned sigmoid function to define p before taking its square root for both gate probability and amplitude. This is a stochastic-construction-to-component relationship, not equivalence to sigmoid or a deterministic-function subclass. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Yoshua Bengio, Nicholas Léonard, Aaron Courville
|
primary paper
full_text
|
ID: arXiv:1308.3432v1
sha256:509eb5e5294d…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T17:00:15Z
Updated: 2026-10-11T17:00:15Z
Source File:
data/curated/StochasticTimesSmoothUnit.yaml