Supermasks in Superposition
Algorithms AIMech:SupSupSupermasksInSuperposition DRAFT
Aliases: SupSup, HopSupSup, Supermasks in Superposition Continual Learning
A continual learning algorithm that eliminates catastrophic forgetting by keeping a shared base neural network fixed at random signed-Kaiming initialization W and learning a sparse binary supermask M^i in {0, 1} for each task i, while inferring unknown task identity at inference (or training) time from a weighted superposition p(alpha) = f(x, W odot sum_{i=1}^k alpha_i M^i) via One-Shot or Binary gradient minimization of output Shannon entropy H(p(alpha)) or superfluous-neuron log-partition objective G(y; alpha) = log sum_v exp(y_v).
Identity Notes & Concept Scope
Introduced by Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi (NeurIPS 2020, pp. 15173-15184, arXiv:2006.14769) as Supermasks in Superposition (SupSup) and HopSupSup. Distinct from LT-SFT (AIMech:LTSFTLotteryTicketSparseFineTuning), which fine-tunes real-valued sparse difference vectors from a pretrained language model and sums them unconditionally; and from Polytropon (AIMech:PolytroponModularSkillAllocation), which jointly learns LoRA skill modules and a binary task-skill allocation matrix.
Claims & Grounded Evidence
4 Claims
Mechanism
#supsup-superposition-entropy-one-shot-and-binary-task-inference
Given k binary supermasks {M^i}_{i=1}^k over fixed signed-Kaiming weights W, SupSup forms the uniform superposition p(alpha) = f(x, W odot sum_{i=1}^k alpha_i M^i) at alpha_i = 1/k and infers unknown task identity either in One-Shot via a single Frank-Wolfe gradient step argmax_i (-dH(p(alpha))/dalpha_i) or in log_2(k) Binary halving steps that zero out the bottom median of -nabla_alpha H(p(alpha)) and renormalize alpha on the simplex.
Mechanism
#supsup-superfluous-neurons-gradient-theorem-and-nns-allocation
Adding superfluous output neurons S = {l+1, ..., s} and defining G(y; alpha) = log sum_{v'=1}^s exp(y_{v'}) with gradients detached on real class neurons R = {1, ..., l} makes dG/dy_v identically match the cross-entropy training gradient dL/dy_v on S (Lemma I.1) so that E[-dG/dalpha_j] > 0 >= E[-dG/dalpha_i] for i != j (Lemmas I.2-I.3), while in task-agnostic Scenario NNs a new supermask is allocated whenever k max_i softmax(-nabla_alpha H)_i < 1 + epsilon.
Application
#supsup-2500-task-permuted-mnist-and-splitimagenet-continual-learning
Because W is fixed and generated from a single random seed, SupSup stores only sparse binary masks (achieving 89.58% vs. BatchE's 81.50% top-1 accuracy on 100-task SplitImageNet with ResNet-50) and sequentially learns 2,500 tasks on PermutedMNIST (25,000-way classification) without catastrophic forgetting, maintaining 93.46% accuracy in Scenario GNu and 93.28% in Scenario NNs using single-image One-Shot task inference.
Limitation
#supsup-miscalibration-and-similar-task-inference-limitation
Gradient-based task identity inference degrades when tasks are closely related (such as 10-degree increments in RotatedMNIST, which require multi-step Binary search over a full batch rather than single-image One-Shot inference) or when uncalibrated subnetworks produce overconfident low-entropy predictions on out-of-task data.
⚠️ Curated Limitations & Negative Results
- One-Shot single-gradient task identity inference can fail when tasks share highly similar input distributions (e.g., 10-degree rotations in RotatedMNIST), requiring log_2(k)-step Binary search and full test batches.
- Entropy-based task inference relies on subnetworks being well-calibrated (producing low-entropy outputs only on their own task's distribution) and degrades when wrong-task masks are overconfident.
- Storing masks implicitly as attractors in a fixed-size Hopfield network (HopSupSup) is quadratic in mask dimension, restricting masking to layer activations rather than all weight parameters.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:SparseLearning
Sparse Learning
|
related_match | SupSup learns sparse binary subnetwork supermasks over a fixed random network and superimposes them for continual learning. |
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | SupSup operates on randomly initialized deep neural networks and Hopfield associative memory networks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r715-primary-pdf
|
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, Ali Farhadi
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=Supermasks+in+Superposition+Wortsman&h=5&format=xml
sha256:9dfbf7bbe5e1…
|
r715-arxiv-abs
|
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, Ali Farhadi
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2006.14769
sha256:182cc2f76562…
|
r715-metadata-cr
|
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, Ali Farhadi
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=Supermasks+in+Superposition+Wortsman&h=5&format=xml
sha256:b833ce75e514…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T13:18:00Z
Updated: 2026-10-19T13:18:00Z
Source File:
data/curated/SupSupSupermasksInSuperposition.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories