Home / Curated / Algorithms / Iterative Product Quantization for Neural Networks (And the Bit Goes Down / Activation-Weighted PQ)

Iterative Product Quantization for Neural Networks (And the Bit Goes Down / Activation-Weighted PQ)

Algorithms AIMech:AndTheBitGoesDownIterativeProductQuantizationForNeuralNetworks DRAFT

Aliases: Iterative Product Quantization for Neural Networks, And the Bit Goes Down Activation-Weighted Product Quantization, Activation-Weighted Product Quantization with Codebook Distillation

Iterative Product Quantization for Neural Networks (introduced in 'And the Bit Goes Down: Revisiting the Quantization of Neural Networks', often denoted iPQ) is a structured vector quantization method that compresses deep neural networks sequentially from lowest to highest layer by partitioning weight columns (or spatially reshaped K x K convolutional filters) into contiguous d-dimensional subvectors, learning a byte-aligned codebook C via a weighted k-means EM algorithm that minimizes the in-domain output activation reconstruction error ||y - y_hat||_2^2 = sum_j ||x(w_j - q(w_j))||_2^2 on unlabelled input activations rather than Frobenius weight error, and fine-tuning the learned codewords both layer-by-layer and globally by averaging subvector gradients under a teacher-student Kullback-Leibler distillation loss.

Identity Notes & Concept Scope Distinct from classical weight-space Product Quantization (Gong et al., 2014), which minimizes Frobenius weight error ||W - W_hat||_2^2 (Eq. 1) and suffers catastrophic accuracy drift across deep layers, and distinct from Quant-Noise (AIMech:QuantNoiseTrainingWithQuantizationNoiseForExtremeCompression), which applies quantization noise to random weight blocks during training to preserve unbiased gradient flow.

Claims & Grounded Evidence

4 Claims

Mechanism #ipq-activation-weighted-kmeans-em-objective

Instead of minimizing Frobenius weight reconstruction error ||W - W_hat||_2^2 (Eq. 1), iPQ minimizes in-domain output activation reconstruction error ||y - y_hat||_2^2 = sum_j ||x(w_j - q(w_j))||_2^2 (Eq. 2) by unrolling input activations x in R^{B x C_in} into x_tilde in R^{(B*m) x d} and alternating an E-step cluster assignment c_j = argmin_{c in C} ||x_tilde(c - v)||_2^2 (Eq. 3) with a least-squares M-step update c* = argmin_{c in R^d} sum_{p in I_c} ||x_tilde(c - v_p)||_2^2 (Eq. 4).

supports Source: And the Bit Goes Down: Revisiting the Quantization of Neural Networks · Locator: Section 3.1 (Quantization of a Fully-Connected Layer), Equations 1-4, pp. 3-4
supports Source: And the Bit Goes Down: Revisiting the Quantization of Neural Networks · Locator: frobenius_weight_error_sq and activation_recon_error_sq

Mechanism #ipq-sequential-layer-quantization-and-codeword-distillation

For convolutional layers W in R^{C_out x C_in x K x K}, iPQ reshapes filters spatially into subvectors of dimension d = K x K (or 2*K*K), quantizes layers sequentially using activations forwarded through already-quantized lower layers, and fine-tunes fixed-assignment codewords under teacher-student KL divergence L = KL(y_s, y_t) via averaged subvector gradients c <- c - eta * (1/|I_c|) sum_{p in I_c} dL/db_p (Eq. 5), followed by global centroid fine-tuning with BatchNorm running-statistics adaptation.

supports Source: And the Bit Goes Down: Revisiting the Quantization of Neural Networks · Locator: Section 3.2-3.4, Figure 2 and Equation 5, pp. 4-5
supports Source: And the Bit Goes Down: Revisiting the Quantization of Neural Networks · Locator: codeword_distillation_step and layer_storage_bytes

Application #ipq-resnet50-and-mask-rcnn-byte-aligned-compression

Using byte-aligned codebooks (k = 256 centroids, 1 byte per subvector index), iPQ compresses a semi-supervised ResNet-50 by 20x to ~5 MB while preserving 76.12% ImageNet top-1 accuracy and compresses Mask R-CNN (ResNet-50 FPN) by 26x from 170 MB to 6.65 MB with 33.9 Box AP and 30.8 Mask AP.

supports Source: And the Bit Goes Down: Revisiting the Quantization of Neural Networks · Locator: Section 4.2-4.3, Tables 1-4 and Figure 3, pp. 6-8

Limitation #ipq-sequential-calibration-time-and-depthwise-conv-limitation

Because each layer's codebook depends on the quantized activations of all preceding layers and requires per-layer distillation fine-tuning (~2,500 iterations per layer), quantizing ResNet-50 takes ~1 GPU day on a V100, and architectures dominated by depthwise separable convolutions (such as MobileNets) exhibit lower spatial filter correlation for PQ than ResNets.

supports Source: And the Bit Goes Down: Revisiting the Quantization of Neural Networks · Locator: Section 2 (Dedicated architectures) and Section 4.1 (Experimental Setup), pp. 3, 6

⚠️ Curated Limitations & Negative Results

Sequential bottom-up layer quantization and per-layer distillation fine-tuning require ~1 V100 GPU day to compress a ResNet-50 model.
Depthwise-separable architectures (e.g., MobileNets, EfficientNets) have less spatial filter redundancy than standard 3x3 ResNet convolutions, reducing PQ gains without joint training.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	broad_match	Activation-weighted Iterative Product Quantization is a vector quantization and codebook distillation algorithm for compressing deep neural networks.
`AIO:Distillation` Distillation	related_match	Uses layer-wise and global Kullback-Leibler knowledge distillation from the uncompressed teacher network to fine-tune PQ codewords on unlabelled data.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r815-primary-pdf`	And the Bit Goes Down: Revisiting the Quantization of Neural Networks Pierre Stock, Armand Joulin, Remi Gribonval, Benjamin Graham, Herve Jegou	primary paper full_text	ID: https://arxiv.org/bibtex/1907.05686 sha256:4e5462e5d025…
`r815-arxiv-abs`	And the Bit Goes Down: Revisiting the Quantization of Neural Networks (arXiv abstract) Pierre Stock, Armand Joulin, Remi Gribonval, Benjamin Graham, Herve Jegou	primary paper abstract_only	ID: https://arxiv.org/abs/1907.05686 sha256:1c6296961624…
`r815-metadata-cr`	And the Bit Goes Down: Revisiting the Quantization of Neural Networks (Crossref metadata) Pierre Stock, Armand Joulin, Remi Gribonval, Benjamin Graham, Herve Jegou	primary paper abstract_only	ID: https://arxiv.org/bibtex/1907.05686 sha256:e3feb828eafe…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T14:58:00Z

Updated: 2026-10-19T14:58:00Z

Source File: data/curated/AndTheBitGoesDownIterativeProductQuantizationForNeuralNetworks.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories