Hierarchical probabilistic neural network language model
Architectures AIMech:HierarchicalProbabilisticNeuralNetworkLanguageModel DRAFT
A neural next-word architecture that factors word probabilities along a binary vocabulary tree and shares a nonlinear branch predictor conditioned on context-word embeddings and a separate embedding of the current tree node.
Identity Notes & Concept Scope
This record covers Morin and Bengio’s 2005 shared-predictor architecture, including its WordNet-guided tree construction. It is a complete language model, distinct from the generic hierarchical-softmax output component and from HLBL’s linear context representation with node-vector scores. The title is scoped to this original construction, not every probabilistic language model using a hierarchy. All 442 pinned AIO labels/scoped synonyms and 355 preceding native IDs/names/aliases were compared, with no normalized identity match. Full-checkout fixed-string searches included ignored and hidden files in both worktree and main, returning 54 lines each. All 41 unique matched contents were read: source references, complete-model contrasts, frontier leads and prior review context for the hierarchical-softmax component. None is an existing record for these complete architectures. Twenty exact AIO nodes and their generated records plus four complete native neighbors were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Concatenate the embeddings of the preceding context words into x, sharing their embedding matrix across all tree nodes. For internal node n, compute q_n=sigmoid(alpha_n+beta^T tanh(c+W x+U N_n)), where N_n is its separately learned node embedding, alpha_n is its scalar bias, and beta, c, W and U are shared parameters. The binary code for a target word chooses q_n or 1-q_n at each node; multiply those factors to obtain its normalized next-word probability. Training backpropagates the observed-word log loss through nodes on that path and their shared parameters. Although W x can be reused, tanh(c+W x+U N_n) generally differs by node. The reported tree starts from WordNet, selects one parent for multiply parented nodes and the most frequent sense per word, then makes nonbinary branches binary using corpus-based clustering. This hierarchy is a modeling choice, not a semantic-equivalence guarantee.
⚠️ Curated Limitations & Negative Results
- Target evaluation work scales with path length and the branch-predictor cost. A logarithmic path requires suitable tree depth; producing the entire vocabulary distribution still requires at least linear output work.
- The learned tree distribution is exactly normalized for its own parameters, but is not an equality-preserving numerical shortcut for a previously specified flat-softmax network.
- Choosing one frequent WordNet sense loses other senses. Multiple leaves per word are discussed as an extension and require summed path probabilities; this record does not attribute that implementation to the original experiment.
- Shared parameters reduce independent per-node parameterization but do not make the hidden activities identical at all nodes or establish convex learning or global convergence.
- The accessed reissue PDF has unresolved references and a page-count discrepancy against its landing-page pagination; evidence is tied to the exact six-page file. No claimed speedup, test ranking or author-code behavior is certified.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:AutoregressiveLanguageModel
Autoregressive Language Model
|
broad_match | The source specifies next-word conditionals given preceding words, defining a sequential language model with a finite context. The AIO target is broader than this shared nonlinear node-conditioned architecture. The mapping uses its sequential-probability scope, not its qualitative assertion about excelling at generation. HierarchicalLanguageModel denotes multilevel representations and is not an exact identity for a vocabulary-tree construction; OutputLayer, SoftmaxLayer and EmbeddingLayer are components rather than whole-model superclasses. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Frederic Morin, Yoshua Bengio
|
primary paper
full_text
|
ID: pmlr-vR5-morin05a
sha256:43409245b14c…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T18:15:42Z
Updated: 2026-10-11T18:15:42Z