Home / Curated / Algorithms / Shake-Shake regularization

Shake-Shake regularization

Algorithms AIMech:ShakeShakeRegularization DRAFT

The original residual-network training and inference procedure that combines two learned branch outputs with complementary uniform random weights during the forward pass, redraws the branch weights for its prescribed backward pass, and uses equal one-half branch weights at inference, with explicitly chosen image-level or minibatch-level coefficient sharing.

Identity Notes & Concept Scope Current creation-context inventory at merged base fc27ccaf4668f1bf7468a10c269f5583fa3e640b covered 742 maintained and 442 pinned AIO records, including IDs, names, aliases and scoped synonyms, with no normalized candidate-probe collision. Four fixed-string word searches included ignored and hidden files and yielded 369/373/1045/1105 occurrences. Root read all eleven distinct matching strings and their selected provenance, while exact reconstruction accounted mechanically for all 2892 occurrences. Hits identify existing frontier leads and historical Fast AutoAugment code inventory paths, not a maintained record for this identity. The newly surfaced shakedrop.py path was listed at repository commit 2424224291616a2158a2265069ee95a78a2d13d9; no implementation was opened or executed. Prior primary-source, selected-neighbor and exact pinned CLASS/incident-edge inspections remain bounded by their individual research acceptances. This is not a global absence or saturation claim. This record denotes the named two-learned-residual-branch regularization procedure with complementary forward coefficients and separately selected prescribed backward coefficients. Its control and sharing configurations are retained within one identity. Stochastic depth instead samples residual survival; ShakeDrop adds a shared Bernoulli ordinary/perturbed switch. The related ResidualNeuralNetwork mapping describes the residual host, not architecture identity or subclass membership.

Claims & Grounded Evidence

6 Claims

Mechanism #two-residual-branches-and-skip

The original core is defined on a residual block with two learned residual transformations F1 and F2 and a skip contribution. The three-branch description counts the skip, rather than three learned residual transformations. The displayed identity-skip baseline is x_next = x + F1(x) + F2(x). Contributions combined by addition must have compatible tensor shapes. The reported dimension-changing blocks use a separate compatible skip construction instead of treating every skip as an identity.

supports Source: Shake-Shake regularization · Locator: Section 1.2, PDF p. 2, Equation 1 and Figure 1; Section 2.1.1, PDF p. 3, dimension-changing skip description.

Mechanism #coupled-uniform-forward-mixture

During the core stochastic forward pass, draw alpha uniformly between zero and one and replace the unweighted residual sum by alpha*F1(x) + (1-alpha)*F2(x), retaining the skip contribution. The two residual weights are coupled through their sum of one. They scale whole branch-output tensors, rather than applying independent elementwise noise or Bernoulli removal to their individual entries.

supports Source: Shake-Shake regularization · Locator: Section 1.2, PDF p. 2, Equation 2 and comparison with drop-path and elementwise noise; Section 2.1.2, PDF p. 3, image-level tensor slicing.

Mechanism #redrawn-backward-coefficients

In the core Shake-Shake configuration, overwrite the scaling coefficients with new random draws before each forward pass and again before each backward pass. Figure 1 distinguishes the forward coefficient alpha from the backward coefficient beta: the two residual branches receive backward scaling beta and 1-beta. The training loss is obtained from the forward realization, while the prescribed branch backward scaling need not reuse its alpha and 1-alpha. This is a deliberately specified backward rule, not an assertion that it is the ordinary chain-rule derivative of that realized forward computation.

supports Source: Shake-Shake regularization · Locator: Section 1.3, PDF p. 2, Figure 1 and training procedure; Section 2.1.2, PDF p. 3, Shake versus Keep definitions.

Mechanism #image-versus-minibatch-sharing

The source distinguishes Batch and Image coefficient sharing for each residual block. Batch uses the same scaling coefficient for all images in the minibatch. Image uses a different coefficient for each image and applies that scalar across the corresponding branch tensor, including all of its channels and spatial positions. Image mode is therefore not independent per-pixel or per-channel coefficient sampling.

supports Source: Shake-Shake regularization · Locator: Section 2.1.2, PDF p. 3, Batch and Image definitions and the image-level update procedure.

Scope #shake-even-keep-controls

Shake means overwriting coefficients with new random numbers before the relevant pass; Even means setting them to one half; Keep means reusing the forward coefficients for the backward pass. These choices distinguish the reported Shake-Shake, Shake-Keep, Shake-Even and Even-Shake configurations. The core redrawn-forward/redrawn-backward method is not made identical to those controls merely because they use the same residual host. Later coefficient-dependence experiments in the paper are configurations, not automatically separate algorithm identities.

supports Source: Shake-Shake regularization · Locator: Section 2.1.2, PDF p. 3, forward/backward procedure definitions; Section 4, PDF pp. 6–7, coefficient-dependence experiments.

Mechanism #half-weight-inference

At inference, replace the random residual scaling coefficients with one half, giving x + (F1(x)+F2(x))/2 for the displayed identity-skip block. This equals the conditional mean of that block’s stochastic forward output when its input and branch outputs are held fixed. It does not recover the unweighted baseline residual sum and does not establish that an entire nonlinear test network equals the expected prediction or loss of its stochastic training realizations.

supports Source: Shake-Shake regularization · Locator: Section 1.2, PDF p. 2, Equations 1–2 and the half-weight test-time instruction. The local conditional-mean qualification follows directly from this uniform-weight formula.

⚠️ Curated Limitations & Negative Results

This record covers the original two-learned-branch core with a compatible skip path. It is not the residual architecture itself, Bernoulli stochastic depth, all drop-path methods, ShakeDrop, or cross-example input/label interpolation such as MixUp. The source also tests architectural variants without a skip or batch normalization in Section 5, PDF pp. 7–8; those experiments do not make every host assumption or every reported setting part of one identical procedure.
Redrawing the backward coefficient generally changes the derivative of the realized forward loss. For scalar branches F1=theta1 and F2=theta2, z=alpha*theta1+(1-alpha)*theta2 and L=z squared/2, the prescribed branch field is (beta*z,(1-beta)*z). With independent uniform alpha and beta its expectation is ((theta1+theta2)/4,(theta1+theta2)/4), whereas the gradient of the expected same forward loss is (theta1/3+theta2/6,theta1/6+theta2/3). This bounded hand example rules out an unconditional unbiased-gradient interpretation of that same objective; it is not an author-issued erratum or proof that the empirical procedure fails.
The half-weight test convention is a local conditional expectation for fixed branch outputs. Nonlinear downstream operations, dependence of intermediate activations on earlier draws and the loss prevent promoting it to an exact ensemble-average theorem. Equal branch outputs can eliminate the forward perturbation for a particular input while separately parameterized branches still receive different backward allocations.
Image, Batch, Shake, Even and Keep choices materially specify the rule. The source’s mathematical uniform draws do not certify a particular pseudorandom generator, endpoint treatment, cross-device synchronization or finite-precision implementation. No later library’s coefficient convention is silently substituted for the inspected procedure.
The source’s decorrelation observations and coefficient/architecture experiments are empirical and scoped. They do not establish universal improvement in generalization, convergence, robustness, runtime or branch independence. The conclusion explicitly leaves the dynamics for further investigation. No benchmark result or author implementation is certified by this record.
Core evidence is the fixed ten-page arXiv version 1705.07485v2, submitted 23 May 2017. The earlier four-page ICLR 2017 workshop manuscript and arXiv v1 are distinct provenance context; PDF build dates are not publication dates and numerical experiments are not merged across versions. Historical supporting research read the full v2 text and selected core visuals; the separately recorded root research reading was selective. The author code was not inspected or executed.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ResidualNeuralNetwork` Residual Neural Network	related_match	The original procedure changes forward training, prescribed backward propagation and inference in a skip-connected host with two learned residual branches. This is a procedure-to-host association, not identity with the residual architecture, an is-a assertion or a requirement that every tested variant preserve every component of the pinned AIO class.

Evidence Sources & Primary Literature

1 Source

Source ID	Publication / Document	Type / Access	Reference & Checksum
`gastaldi2017v2`	Shake-Shake regularization Xavier Gastaldi	primary paper full_text	ID: arXiv:1705.07485v2 sha256:5475784fc5c0…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-15T08:18:23Z

Updated: 2026-10-15T08:18:23Z

Source File: data/curated/ShakeShakeRegularization.yaml