Temporal factored restricted Boltzmann machine language model
Architectures AIMech:TemporalFactoredRestrictedBoltzmannMachineLanguageModel DRAFT
Aliases: Temporal FRBM
A sequential extension of the factored RBM language model that directs information from the preceding binary hidden state into the current hidden bias, propagating context beyond the explicit word window.
Identity Notes & Concept Scope
This is the temporal FRBM in Mnih and Hinton’s 2007 language-model paper. It extends the factored categorical-word construction with shared hidden-to-hidden weights A. It is distinct from the static FRBM, the broader temporal RBM with additional visible-history connections, and deterministic recurrent language models. Its approximate filtering algorithm is not the exact latent-state model. All 442 pinned AIO labels/scoped synonyms and 358 preceding native IDs/names/aliases were compared, with no normalized identity match. Full-checkout fixed-string searches included ignored and hidden files in both worktree and main, returning 33 lines each. All 24 unique matched contents were read: LBL source references and model contrasts, discovery leads and saved review context. These do not constitute existing candidate records. All 85 semantic label/synonym hits and 22 exact AIO nodes were inspected, alongside corresponding generated records and four complete native neighbors. Previously read contextual files were checked for byte identity. Absence is bounded to this snapshot and inspected neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Apply shared FRBM parameters to successive overlapping word windows. Connect hidden state h_tau to h_(tau+1) so that the current hidden bias becomes b_h+A h_tau; conditional on that earlier state, the current word and hidden variables retain an FRBM joint with its own normalizer. Exact prediction must account for uncertainty in previous hidden states. The proposed online approximation instead substitutes their inferred probabilities p_tau, computes p_(tau+1)=sigmoid(b_h+A p_tau+sum_i W_i^T r_word_i) after observing the current window, and predicts the next word using the FRBM word distribution with the correspondingly shifted bias. Learning uses these approximate filtered probabilities, applies contrastive-divergence updates to individual model instances, and combines them. The temporal-weight update contrasts current hidden probabilities and their reconstruction, multiplied by the preceding filtered vector; it does not propagate an exact likelihood gradient through the full latent history.
⚠️ Curated Limitations & Negative Results
- Replacing an uncertain earlier hidden state by its mean inside a sigmoid or a context-dependent normalized word distribution is generally not exact marginalization. Observing a new word can also change the posterior over previous hidden states.
- The cited temporal-RBM source explicitly ignores future evidence for online learning and notes that changes in its parameter-dependent approximate posterior can decrease the variational bound. Approximate filtering plus finite CD therefore does not guarantee monotonic likelihood or bound improvement.
- The recurrent path permits dependence beyond the explicit context window, but does not guarantee faithful long-range memory, tractable exact filtering or a particular effective context length.
- This construction has directed temporal links and undirected current word–hidden interactions. It is not equivalent to a single unconditioned RBM, a pure directed Bayesian network parameterization, or a deep belief network merely because latent states are present.
- The static FRBM marginal’s printed bias-sign conflict remains qualified when its conditional prediction is reused. No implementation audit, empirical superiority or exact sequence-likelihood result is asserted.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:AutoregressiveLanguageModel
Autoregressive Language Model
|
broad_match | The primary construction predicts the next word from preceding words, defining sequential word conditionals. The pinned AIO target is broader than this temporally linked factored latent-word architecture. This mapping uses sequential-probability scope, not an empirical generation-quality claim. FactoredLanguageModel denotes explicit linguistic factors, not factorized neural weight matrices; ImplicitLanguageModel and EnergyBasedLLM describe non-autoregressive sequence energies rather than these next-word conditionals. |
AIO:RestrictedBoltzmannMachine
Restricted Boltzmann Machine
|
related_match | The architecture uses an undirected RBM interaction between the current categorical word and binary hidden state, with context affecting hidden inputs. A related mapping records that explicit component/conditional construction without asserting that the entire autoregressive or temporally linked sequence model is one unconditioned RBM. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Andriy Mnih, Geoffrey Hinton
|
primary paper
full_text
|
ID: 10.1145/1273496.1273577
sha256:d0709a403248…
|
temporal-background
|
Ilya Sutskever, Geoffrey Hinton
|
primary paper
full_text
|
ID: pmlr-v2-sutskever07a
sha256:8b8b19704c75…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T18:27:33Z
Updated: 2026-10-11T18:27:33Z