Stochastic depth
Algorithms AIMech:StochasticDepth DRAFT
A residual-network training procedure that randomly bypasses residual transformations during training and uses all residual transformations with survival-probability scaling during inference.
Identity Notes & Concept Scope
This record covers the original 2016 stochastic-depth procedure with a block-survival draw per minibatch and test-time residual scaling. It is a training procedure for residual networks, not the residual architecture itself. Modern per-example drop-path functions, inverted scaling conventions and learned dynamic-depth methods require separate version assessment. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 276 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in worktree and main, returning 5 and 5 lines respectively. Every text hit was a prior candidate lead or a closely related variant lead. Nine exact AIO CLASS nodes with scoped synonyms/comments and outgoing edges and three native neighboring mechanisms were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Assign each residual block a survival probability and, for each training minibatch, sample whether to execute its residual transformation. An active block adds its residual output to the skip path before the architecture’s output activation; an inactive block bypasses the residual transformation while preserving the skip path. The source’s identity reduction relies on its nonnegative activations and compatible skip connection. It proposes a linear decrease in survival probability with block depth from an always-present input to a chosen last-block probability, while also considering a uniform assignment. The expected number of active residual blocks is the sum of their survival probabilities. During inference, activate every residual transformation and multiply its output by its training survival probability before adding the skip path. The original convention does not divide active training residuals by the survival probability.
⚠️ Curated Limitations & Negative Results
- Identity bypass is conditional on the skip path and activation: a post-addition ReLU clips negative skip values, and a dimension-changing projection is not the same operation as an identity map. The procedure cannot be transferred to an arbitrary incompatible block by merely deleting its output.
- The survival schedule in the core method varies with network depth, not automatically with elapsed training time. The paper’s final-block probability of0.5 is an experimental choice. Mini-batch sampling and inference scaling differ from later per-example or inverted-scaled implementations.
- Survival-weighted inference is a practical combination of shared-weight subnetworks, not generally the exact expected prediction of the stochastic nonlinear network. Reduced expected active-block count does not by itself guarantee a proportional wall-clock saving; computation must actually be bypassed.
- The source’s empirical gradient and benchmark observations do not certify absence of vanishing gradients, universal regularization improvement or arbitrarily deep trainability. No author code or trained network was executed.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ResidualNeuralNetwork
Residual Neural Network
|
related_match | The original procedure trains a residual network by stochastically bypassing its residual transformations. The AIO target is the network architecture, while the native concept is a training/inference procedure. |
AIO:TrainingStrategy
TrainingStrategy
|
related_match | Stochastic depth changes network participation during training and the corresponding inference convention. The target’s preprocessing genus and broad learning-task examples do not establish a clean subclass boundary, so the association is related. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q. Weinberger
|
primary paper
full_text
|
ID: arXiv:1603.09382v3
sha256:f8c7313da059…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T12:35:00Z
Updated: 2026-10-11T12:35:00Z
Source File:
data/curated/StochasticDepth.yaml