FISH Mask
Algorithms AIMech:FISHMask DRAFT
Aliases: Fisher-Induced Sparse uncHanging Mask, FISH Mask Training, Fisher-Induced Fixed Sparse Mask, Top-k Fisher Sparse Update Mask
A sparse gradient-based neural network training and parameter-efficient transfer learning algorithm (Fisher-Induced Sparse uncHanging Mask) that pre-computes a fixed binary parameter mask before training (or at coarse epoch intervals) and updates only the selected k parameters over many subsequent gradient descent iterations. To measure how strongly each parameter theta_i affects the model output distribution p_theta(y|x) under local KL divergence E_x[D_KL(p_theta(y|x) || p_{theta+delta}(y|x))] = delta^T F_theta delta + O(delta^3), FISH Mask estimates the diagonal Fisher information F_hat_theta = (1/N) sum_{i=1}^N E_{y ~ p_theta(y|x_i)} [(grad_theta log p_theta(y|x_i))^2] (or its supervised empirical Fisher variant (1/N) sum_{i=1}^N (grad_theta log p_theta(y_i|x_i))^2) over N examples and constructs the fixed mask {theta_i | F_hat_{theta_i} >= sort(F_hat_theta)_k} corresponding to a target mask sparsity level.
Identity Notes & Concept Scope
Distinct from Diff Pruning (AIMech:DiffPruning), which optimizes a trainable stretched Hard-Concrete mask over an additive diff vector throughout training at the cost of extra memory and 1.5x-2x slower minibatches, and from BitFit (AIMech:BitFit), which heuristically updates only bias vectors (and cannot be applied to bias-free architectures such as T5). Also distinct from Fisher weight pruning methods, which zero out weights to sparsify the network rather than keeping all pre-trained weights intact while restricting gradient updates to the top-k Fisher coordinates.
Claims & Grounded Evidence
3 Claims
Mechanism
#fish-mask-diagonal-fisher-kl-sensitivity-selection
FISH Mask approximates parameter importance via the diagonal Fisher information F_hat_theta = (1/N) sum_{i=1}^N E_{y ~ p_theta(y|x_i)}[(grad_theta log p_theta(y|x_i))^2] (or the empirical Fisher using ground-truth labels y_i), which governs the second-order KL divergence E_x[D_KL(p_theta(y|x) || p_{theta+delta}(y|x))] = delta^T F_theta delta + O(delta^3), and selects the fixed subset {theta_i | F_hat_{theta_i} >= sort(F_hat_theta)_k} using as few as N = 32 to 1024 samples before gradient descent begins.
Application
#fish-mask-glue-transfer-distributed-and-checkpoint-efficiency
On BERT_LARGE across the GLUE test server, FISH Mask updating only 0.5% of parameters per task achieves an 82.6 average score—matching 100% dense fine-tuning (82.5) and outperforming Diff Pruning at 0.5% (81.5), BitFit at 0.08% (81.2), and a 0.5% Random Mask (76.8)—while reducing ResNet-34 CIFAR-10 checkpoint disk storage by 5x at 10% mask sparsity (93.9% accuracy when refreshed every epoch, matching standard training).
Limitation
#fish-mask-from-scratch-warmup-and-extreme-sparsity-limitation
When training ResNet-34 from scratch on CIFAR-10, fixing a single FISH Mask across the entire training run without periodic refresh or 5 epochs of dense warmup degrades validation accuracy (e.g., dropping to 78.5% at 0.5% fixed mask sparsity vs. 93.9% standard training), and segmenting the FISH Mask into non-overlapping worker partitions in distributed BERT fine-tuning underperforms sharing a single global FISH Mask across workers.
⚠️ Curated Limitations & Negative Results
- From-scratch training requires an initial dense warmup phase (e.g., 5 epochs on CIFAR-10) and periodic mask recomputation (e.g., every epoch) at higher mask sparsity (10%), as keeping a 0.5% mask fixed from random initialization drops accuracy from 93.9% to 78.5% (Section 4.2-4.3, Table 2).
- Storing both 32-bit float values and 32-bit integer coordinate indices only reduces checkpoint or communication size when the mask sparsity level is below 50%, and partitioning the FISH Mask into disjoint worker subsets degrades distributed convergence (Section 4.3, p. 9; Appendix A, p. 14).
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:SparseLearning
Sparse Learning
|
narrow_match | FISH Mask is a sparse learning method that restricts gradient updates to a fixed top-k parameter subset selected via diagonal Fisher information. |
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | FISH Mask trains and fine-tunes deep neural networks with reduced checkpoint storage and distributed worker communication. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r695-primary-pdf
|
Yi-Lin Sung, Varun Nair, Colin Raffel
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=Training+Neural+Networks+Fixed+Sparse+Masks+Sung&format=xml
sha256:4415f2f79ad1…
|
r695-arxiv-abs
|
Yi-Lin Sung, Varun Nair, Colin Raffel
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2111.09839
sha256:48785d5ebf91…
|
r695-metadata-cr
|
Yi-Lin Sung, Varun Nair, Colin Raffel
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=Training+Neural+Networks+Fixed+Sparse+Masks+Sung&format=xml
sha256:8b879bcbc94b…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T12:58:00Z
Updated: 2026-10-19T12:58:00Z
Source File:
data/curated/FISHMask.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories