Edge-Popup Untrained Subnetwork Score Optimization
Algorithms AIMech:EdgePopupUntrainedSubnetworkScoreOptimization DRAFT
Aliases: Edge-Popup, edge-popup Algorithm, Edge-Popup Supermask Optimization, Popup Score Top-k Untrained Subnetwork Search
A subnetwork discovery algorithm for randomly weighted neural networks that keeps all weights w_{uv} (including first, last, and batch-normalization layers) frozen at their random initialization, associates each edge (u, v) with a positive real-valued popup score s_{uv}, deterministically selects the subgraph of edges in each layer l with the top-k% highest scores via hard indicator h(s_{uv}) in {0, 1} to compute layer inputs I_v = sum_u w_{uv} Z_u h(s_{uv}), and updates all scores (including currently inactive edges) via stochastic gradient descent with the straight-through estimator s_tilde_{uv} = s_{uv} - alpha (dL/dI_v) w_{uv} Z_u.
Identity Notes & Concept Scope
Distinct from stochastic Bernoulli supermask sampling (Zhou et al., NeurIPS 2019), which samples binary masks stochastically on every forward pass and suffers high gradient variance at scale, from Discovering Neural Wirings (DNW, Wortsman et al., NeurIPS 2019), which selects top-k% edges by weight magnitude while modifying the weights themselves, from Piggyback (AIMech:PiggybackBinaryWeightMaskMultiTaskAdaptation), which thresholds masks on ImageNet-pretrained backbones with a trained linear classifier head, and from SupSup (AIMech:SupSupSupermasksInSuperposition), which extends Edge-Popup to multi-task superposition and one-shot entropy-based task inference.
Claims & Grounded Evidence
4 Claims
Mechanism
#r721-claim-swap-descent-theorem
Under the Edge-Popup straight-through score update s_tilde_{uv} = s_{uv} - alpha (dL/dI_v) w_{uv} Z_u, whenever a non-empty set of inactive edges E_new whose weighted outputs w_{uv} Z_u align with the negative loss gradient -dL/dI_v increases in score enough to replace an equal number of edges E_old in a layer's top-k% active set (s_{new} < s_{old} and s_tilde_{new} > s_tilde_{old}), the first-order Taylor expansion guarantees that the mini-batch loss strictly decreases: L(I_tilde) < L(I).
Mechanism
#r721-claim-combinatorial-search-space-and-scaling
Because Edge-Popup selects kn weights out of n per layer, the combinatorial search space of (n choose kn) subnetworks is maximized when k is in [30%, 70%] (peaking near k = 50% while degenerating to empty networks as k -> 0% and random untrained networks as k -> 100%), and sampling weights from the Signed Kaiming Constant distribution U_k = {-sigma_k, +sigma_k} with sigma_k = sqrt(2 / n_{l-1}) (or Scaled Kaiming with factor sqrt(1/k)) preserves forward activation variance and substantially outperforms Xavier Normal or Kaiming Uniform initialization.
Application
#r721-claim-imagenet-untrained-subnetwork-application
Without ever modifying any weight or batch-normalization parameter, Edge-Popup discovers an untrained subnetwork (k = 30%, 20.6M parameters) inside a randomly weighted Wide ResNet-50 initialized with Signed Kaiming Constant that achieves 73.3% top-1 accuracy on ImageNet—matching a dense 21.8M-parameter ResNet-34 trained with SGD (73.3%)—and discovers a 13M-parameter untrained subnetwork inside ResNet-101 achieving 72.3% top-1 accuracy.
Limitation
#r721-claim-not-lottery-tickets-limitation
Subnetworks discovered by Edge-Popup at random initialization are not Lottery Ticket winning tickets: when their weights are subsequently trained from initialization with SGD on ImageNet, they do not reach the accuracy of the dense parent network and perform no better than training a randomly chosen 30% subnetwork (e.g., 73.6% vs. 73.5% on ResNet-50), and finding competitive untrained subnetworks requires the parent network to be substantially wider and deeper than a standard trained model.
⚠️ Curated Limitations & Negative Results
- Requires an overparameterized parent network with substantially greater width or depth than a standard weight-trained network to match dense trained accuracy, and fails when k approaches 0% or 100% or when weights are initialized with Xavier Normal or Kaiming Uniform distributions.
- Discovered subnetworks do not exhibit the Lottery Ticket property when their weights are subsequently trained with SGD, achieving roughly the same accuracy as training a random subnetwork of equal sparsity.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | Edge-Popup optimizes binary subnetwork connectivity scores within fixed randomly initialized deep neural networks without updating any weight values. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r721-primary-pdf
|
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=What%27s+Hidden+in+a+Randomly+Weighted+Neural+Network+Ramanujan&format=xml
sha256:f5a836023e8c…
|
r721-arxiv-abs
|
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1911.13299
sha256:9e48095891c2…
|
r721-metadata-cr
|
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=What%27s+Hidden+in+a+Randomly+Weighted+Neural+Network+Ramanujan&format=xml
sha256:ef11c49e4714…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T13:24:00Z
Updated: 2026-10-19T13:24:00Z
Source File:
data/curated/EdgePopupUntrainedSubnetworkScoreOptimization.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories