DoReFa-Net (Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients)
Algorithms AIMech:DoReFaNetTrainingLowBitwidthConvolutionalNeuralNetworksWithLowBitwidthGradients DRAFT
Aliases: DoReFa-Net, DoReFa-Net Low-Bitwidth Gradient Quantization, Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
DoReFa-Net (Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients) is a low-bitwidth neural network training and inference framework that executes both forward and backward passes via bitwise AND and popcount convolution kernels (Eq. 3) by deterministically quantizing weights to k_w bits via tanh-normalized uniform quantization f_omega^k(r_i) = 2 * quantize_k(tanh(r_i) / (2 * max(|tanh(r_i)|)) + 0.5) - 1 (or layer-mean scaling E(|r_i|) * sign(r_i) for 1-bit weights), deterministically quantizing [0, 1]-bounded activations to k_a bits via f_alpha^k(r) = quantize_k(r), and stochastically quantizing backpropagated activation gradients dr to k_g bits via per-instance max-abs scaling and additive uniform noise N(k) = sigma / (2^k - 1) with sigma ~ Uniform(-0.5, 0.5).
Identity Notes & Concept Scope
Distinct from PACT (AIMech:PACTParameterizedClippingActivationForQuantizedNeuralNetworks) which learns a parameterized activation clipping bound alpha with full-precision gradients rather than fixed [0, 1] clipping with stochastic low-bitwidth gradient quantization, and distinct from TTQ (AIMech:TTQTrainedTernaryQuantization) which learns asymmetric ternary weight scales {-W_l^n, 0, +W_l^p}.
Claims & Grounded Evidence
4 Claims
Mechanism
#dorefa-net-bitwise-dot-product-and-tanh-weight-activation-quantization
DoReFa-Net decomposes fixed-point dot products between M-bit and K-bit vectors into bitwise AND and bitcount operations sum_{m=0}^{M-1} sum_{k=0}^{K-1} 2^{m+k} bitcount[and(c_m(x), c_k(y))] (Eq. 3), deterministically quantizes k-bit weights via f_omega^k(r_i) = 2 * quantize_k(tanh(r_i) / (2 * max(|tanh(r_i)|)) + 0.5) - 1 (Eq. 9; or sign(r_i) * E(|r_i|) for k = 1, Eq. 7), and quantizes [0, 1]-clipped activations via f_alpha^k(r) = quantize_k(r) (Eq. 11) with Straight-Through Estimator backpropagation (Eqs. 5-6).
Mechanism
#dorefa-net-stochastic-unbiased-gradient-quantization-and-maxpool-fusion
Because backpropagated activation gradients dr are unbounded and span a wide dynamic range, DoReFa-Net maps dr into [0, 1] per sample via max_0(|dr|), injects uniform dithering noise N(k) = sigma / (2^k - 1) with sigma ~ Uniform(-0.5, 0.5) to yield unbiased k-bit stochastic gradients f_gamma^k(dr) = 2 * max_0(|dr|) * [quantize_k(dr / (2 * max_0(|dr|)) + 0.5 + N(k)) - 0.5] (Eqs. 12-14), and commutes activation quantization past max-pooling via quantize_k(max(a, b)) = max(quantize_k(a), quantize_k(b)) (Eq. 15).
Application
#dorefa-net-1bit-weight-2bit-activation-6bit-gradient-imagenet-training
On SVHN, CIFAR-10, and ImageNet (AlexNet), DoReFa-Net trains convolutional networks with 1-bit weights, 2-bit activations, and 6-bit stochastic gradients (or 8-bit gradients with <4-bit weights/activations) while maintaining competitive classification accuracy and enabling bitwise acceleration on CPU, FPGA, and ASIC platforms.
Limitation
#dorefa-net-first-last-layer-full-precision-and-sub-4bit-gradient-collapse
DoReFa-Net keeps the first convolutional layer and final classification layer unquantized because quantizing them significantly degrades accuracy (Section 2.6), and quantizing backpropagated gradients below 4 bits (e.g., 2-bit gradients) causes substantial accuracy degradation.
⚠️ Curated Limitations & Negative Results
- Requires leaving the first convolutional layer and the final classification layer in higher precision to avoid substantial accuracy drops.
- Backpropagated gradient quantization below 4 bits causes significant accuracy degradation even with stochastic dithering.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
broad_match | DoReFa-Net is a low-bitwidth quantization method for training and inference of deep convolutional neural networks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r822-primary-pdf
|
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, Yuheng Zou
|
primary paper
full_text
|
ID: https://arxiv.org/bibtex/1606.06160
sha256:e38d5092f8fa…
|
r822-arxiv-abs
|
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, Yuheng Zou
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1606.06160
sha256:e5447e03e437…
|
r822-metadata-cr
|
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, Yuheng Zou
|
primary paper
abstract_only
|
ID: https://arxiv.org/bibtex/1606.06160
sha256:f1bc6f277d99…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T15:05:00Z
Updated: 2026-10-19T15:05:00Z
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories