Factored restricted Boltzmann machine language model
Architectures AIMech:FactoredRestrictedBoltzmannMachineLanguageModel DRAFT
Aliases: FRBM
A conditional neural word model that factorizes position-specific word-to-hidden weights through a shared word-embedding matrix and couples the next categorical word to binary hidden units in a context-conditioned restricted Boltzmann machine.
Identity Notes & Concept Scope
This is Mnih and Hinton’s static FRBM language architecture. Factored refers to matrices G_i=R W_i with shared learned word features, not explicit part-of-speech or morphological factors. Its context words are conditioned on; their marginal distribution is not learned by this single conditional. Temporal FRBM adds directed hidden-state history, while LBL removes stochastic hidden variables and Replicated Softmax models an undirected document of a given length. All 442 pinned AIO labels/scoped synonyms and 358 preceding native IDs/names/aliases were compared, with no normalized identity match. Full-checkout fixed-string searches included ignored and hidden files in both worktree and main, returning 33 lines each. All 24 unique matched contents were read: LBL source references and model contrasts, discovery leads and saved review context. These do not constitute existing candidate records. All 85 semantic label/synonym hits and 22 exact AIO nodes were inspected, alongside corresponding generated records and four complete native neighbors. Previously read contextual files were checked for byte identity. Absence is bounded to this snapshot and inspected neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Write r_w for the shared column embedding of word w. For context positions 1 through n-1 and candidate word w at position n, define t=b_h+sum_(i<n) W_i^T r_context_i+W_n^T r_w and candidate bias a_w=b_r^T r_w+b_w. The conditional joint energy is E(w,h;context)=-t^T h-a_w. Summing binary h gives unnormalized word mass exp(a_w) times product_j(1+exp(t_j)); normalize these masses over candidate words. Given all words, hidden units are independent Bernoulli with probabilities sigmoid(t_j). Given h, the next word has softmax scores r_w^T W_n h+a_w. Exact conditional likelihood gradients use data-posterior statistics minus model expectations, whose vocabulary sum is explicit. The paper instead trains latent-variable models using finite contrastive divergence: start at the observed word, update hidden, word and hidden states, and contrast resulting statistics. Some runs replace the sampled one-hot word with its probability vector, a mean-field reconstruction variant.
⚠️ Curated Limitations & Negative Results
- Equation (5) prints negative candidate-bias terms, whereas exp(-E) from Equations (2)–(4) and the word conditional in Equation (9) require positive a_w. The curated marginal follows those defining equations and discloses the sign conflict; it does not infer author-code behavior.
- The conditional partition is tractable by summing the vocabulary after binary-hidden marginalization. Generic RBM partition-function intractability must not be copied unchanged into this clamped-context construction; the exact calculation can still be expensive for a large vocabulary.
- Finite-step contrastive divergence is not the exact conditional maximum-likelihood gradient. A deterministic probability-vector reconstruction is not a categorical Gibbs sample.
- Sharing word features across positions does not make sequence context permutation invariant because W_i depends on position. Low-rank factorization constrains the effective interaction matrices.
- No predictive ranking, semantic quality guarantee, global training convergence or implementation certification is asserted.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:AutoregressiveLanguageModel
Autoregressive Language Model
|
broad_match | The primary construction predicts the next word from preceding words, defining sequential word conditionals. The pinned AIO target is broader than this shared low-rank, context-conditioned latent-word architecture. This mapping uses sequential-probability scope, not an empirical generation-quality claim. FactoredLanguageModel denotes explicit linguistic factors, not factorized neural weight matrices; ImplicitLanguageModel and EnergyBasedLLM describe non-autoregressive sequence energies rather than these next-word conditionals. |
AIO:RestrictedBoltzmannMachine
Restricted Boltzmann Machine
|
related_match | The architecture uses an undirected RBM interaction between the current categorical word and binary hidden state, with context affecting hidden inputs. A related mapping records that explicit component/conditional construction without asserting that the entire autoregressive or temporally linked sequence model is one unconditioned RBM. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Andriy Mnih, Geoffrey Hinton
|
primary paper
full_text
|
ID: 10.1145/1273496.1273577
sha256:d0709a403248…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T18:27:33Z
Updated: 2026-10-11T18:27:33Z