Replicated Softmax
Architectures AIMech:ReplicatedSoftmax DRAFT
An undirected document model with binary hidden units and a length-dependent set of categorical word units that share visible-to-hidden weights, with hidden biases scaled by document length.
Identity Notes & Concept Scope
This is the original Replicated Softmax model of Salakhutdinov and Hinton. It specializes a restricted Boltzmann machine to word-count sufficient statistics across documents of different lengths. It is distinct from repeatedly applying the ordinary softmax function, from the expected-count Constrained Poisson update discussed in the paper, and from the directed deterministic-feature Document NADE architecture. All 442 pinned AIO labels/scoped synonyms and 352 preceding native IDs/names/aliases were compared, with no normalized identity match. Both full-checkout searches included ignored and hidden files; worktree returned 15 lines and main 19. All 16 unique matched contents were read: DocNADE contrasts, frontier and review references, plus four ignored-environment Sage/Pygments entries unrelated to AI SAGE. Twenty-one exact AIO nodes and four complete native neighbors were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
For a document with D word positions, vocabulary counts n_k and binary hidden states h_j, use energy E(V,h)=-sum_jk W_jk h_j n_k-sum_k b_k n_k-D sum_j a_j h_j. The joint probability is proportional to exp(-E) with a partition function for that length. Hidden units are conditionally independent given words, with P(h_j=1|V)=sigmoid(D a_j+sum_k W_jk n_k). Given h, positions independently draw words from softmax(b_k+sum_j W_jk h_j); equivalently, sample D word counts from the corresponding multinomial. Parameter sharing makes the ordered-word probability permutation invariant. A count-vector probability includes the number of distinct ordered realizations. Learning contrasts data and model sufficient statistics; the paper approximates the model term with finite alternating Gibbs steps initialized at data.
⚠️ Curated Limitations & Negative Results
- Tractable hidden posteriors do not make the partition function or equilibrium model expectation tractable. Finite-step contrastive-divergence updates are not exact maximum-likelihood gradients; deterministic expected counts are not the same as multinomial Gibbs sampling.
- The model is conditional on document length and supplies no document-length distribution. Its hidden feature vector has independent conditional Bernoulli coordinates, not topic proportions constrained to sum to one.
- For ordinary counting measure on K^D ordered word assignments and F binary hidden units, the zero-weight, zero-bias partition is K^D times 2^F. The AIS discussion’s printed Z_0=2^F needs a corresponding normalized base-measure convention or visible-state factor; it is not adopted unchanged as that counting-measure partition. This is a mathematical qualification of the displayed derivation, not a claim about author-code behavior.
- No convergence, topic quality or benchmark advantage is asserted. The later experimental pages and author software were not audited.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:RestrictedBoltzmannMachine
Restricted Boltzmann Machine
|
broad_match | The inspected local model is explicitly an RBM with categorical visible units, shared word parameters and length-scaled hidden biases. The pinned AIO RBM class is broader and does not specify those document constraints. |
AIO:ProbabilisticTopicModel
Probabilistic Topic Model
|
related_match | The primary source develops an undirected topic/document model. The AIO target belongs to a task hierarchy and includes temporal theme analysis; a related match preserves that context without imposing a task-to-architecture subclass or temporal mechanism. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Ruslan Salakhutdinov, Geoffrey Hinton
|
primary paper
full_text
|
ID: https://proceedings.neurips.cc/paper/2009/hash/31839b036f63806cba3f47b93af8ccb5-Abstract.html
sha256:b1dfdf333f8d…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T18:02:59Z
Updated: 2026-10-11T18:02:59Z
Source File:
data/curated/ReplicatedSoftmax.yaml