Stochastic Pooling

For regional activations a_i>=0 with positive total, compute p_i=a_i/sum_j a_j, draw one categorical location, and output the selected activation. Positive nonmaximal entries can be selected; zero-probability entries cannot. The displayed rule is undefined when every activation is zero, and the paper does not specify a sampling fallback. Setting a zero output in that case would not by itself determine the routing convention. The source uses rectified activations and spatial regions, not a maximum over learned affine channels.

The described backward pass routes the gradient through the same location sampled in the forward pass. Treating that location as fixed omits the derivative of the activation-dependent sampling probabilities; it is not automatically an unbiased gradient of the expected sampled objective. Independent random draws conditional on current activations also do not imply unconditional independence across layers, because earlier selections change deeper activations and their probabilities. Finite training does not guarantee use of every positive entry.

Deterministic inference replaces a regional sample by sum_i p_i*a_i=sum_i a_i^2/sum_i a_i. For nonnegative positive-total inputs this is the exact conditional local first moment and lies between the ordinary arithmetic mean and the maximum. It is generally neither ordinary mean pooling nor the maximum. Substituting local means through nonlinear downstream computations or subsequent activation-dependent pooling generally differs from averaging complete stochastic network predictions. The n^d count for d regions of n locations is a count of possible routing settings, not independently trained models or a guarantee that all settings have positive probability.

The source uses three convolutional layers with 5x5 filters and 64 maps each, rectifiers, 3x3 pooling at stride 2, response normalization and a softmax classifier, trained for 280 epochs. CIFAR-10 has 50,000 training images; 5,000 are used for cross-validation of learning rates and weight decay, with the latter set to 0.001. Pixels are scaled to [0,1] and the per-pixel mean is subtracted. Table 5 does not state the final validation/refit membership, test-set size, repeated-run variability or error bars. The CIFAR-10 inference comparison averages output probabilities across stochastic forward passes, which is arithmetic predictive averaging. Table 5 reports 15.20 percent test error for stochastic training with deterministic probability weighting and 15.12 percent for 100 sampled passes, despite its caption describing probability weighting as best. Those finite classification errors do not establish identical predictions or exact full-network expectation. Training with deterministic probability weighting is a different procedure and reports 19.40 percent error with the same inference choice. The small numerical difference is not a demonstrated statistically significant advantage.

The experiments evaluate configured convolutional classifiers on MNIST, CIFAR-10, CIFAR-100 and SVHN, with rectifiers, response normalization, weight decay and specified training schedules. Region size, stride, architecture and optimization remain choices even though the probability rule adds no learned parameter. Different widths and training durations in the SVHN configurations prevent a width-independent performance conclusion. Saved-switch deconvolutional pictures are qualitative visualizations, not an exact inverse or information-preservation proof. The results do not establish universal regularization benefit, negligible cost on every implementation or current superiority.