Neural probabilistic language model
Architectures AIMech:NeuralProbabilisticLanguageModel DRAFT
Aliases: NPLM
A neural finite-context language architecture that jointly learns a shared word-embedding lookup and a nonlinear predictor of the next-word distribution using concatenated context embeddings and a vocabulary softmax.
Identity Notes & Concept Scope
This record is scoped to Bengio, Ducharme, Vincent and Jauvin’s published 2003 architecture, especially its tanh feed-forward formulation with optional direct input-to-output weights. It does not designate every neural probabilistic model of language, or assert historical priority over the earlier work acknowledged by the paper. It differs from LBL’s tied bilinear output embeddings and Morin–Bengio’s tree-node-conditioned nonlinear predictor. All 442 pinned AIO labels/scoped synonyms and 358 preceding native IDs/names/aliases were compared, with no normalized identity match. Full-checkout fixed-string searches included ignored and hidden files in both worktree and main, returning 33 lines each. All 24 unique matched contents were read: LBL source references and model contrasts, discovery leads and saved review context. These do not constitute existing candidate records. All 85 semantic label/synonym hits and 22 exact AIO nodes were inspected, alongside corresponding generated records and four complete native neighbors. Previously read contextual files were checked for byte identity. Absence is bounded to this snapshot and inspected neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Look up each preceding word in a shared matrix C and concatenate the n-1 context embeddings in their specified positions into x. Compute vocabulary logits y=b+W x+U tanh(d+H x), where direct weights W may be set to zero. Apply the vocabulary softmax and multiply next-word conditionals to obtain sequence probabilities under the chosen context convention. Jointly optimize the embeddings and network parameters by stochastic gradient ascent on the corpus log likelihood with regularization; a weight-decay term added to a maximized objective has the negative penalty sign. The input embedding lookup is shared across context positions, while the output matrices and biases are separately learned parameters. Contributions from repeated appearances of the same context word accumulate in its embedding row. The paper also discusses mixing the resulting predictions with an interpolated trigram model, which is an additional ensemble choice.
⚠️ Curated Limitations & Negative Results
- The specified feed-forward formulation has a finite context window and a fixed vocabulary. It is not a recurrent state model or a tree-normalized output architecture.
- The flat vocabulary softmax incurs vocabulary-dependent computation even when only the observed next-word likelihood is required. Joint learning is nonconvex and does not guarantee optimality or universally meaningful semantic distances.
- An input-embedding row absent from an example’s context has zero gradient from that example’s likelihood, even if its word is the target. This statement excludes regularization, which can still affect unused rows. Output-word parameters remain separate and can receive gradients.
- Sharing the lookup matrix does not discard context order: concatenation places embeddings in different slots with distinct subsequent weights. Optional direct weights change the parameterization but not probability normalization.
- The paper warns that regularizing W and H without C can permit a compensating growth of embedding scale. Smooth feature sharing motivates generalization, but does not prove correct probabilities for every unseen combination. No benchmark superiority or author-code behavior is certified.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:AutoregressiveLanguageModel
Autoregressive Language Model
|
broad_match | The primary construction predicts the next word from preceding words, defining sequential word conditionals. The pinned AIO target is broader than this jointly learned embedding and nonlinear flat-output architecture. This mapping uses sequential-probability scope, not an empirical generation-quality claim. FactoredLanguageModel denotes explicit linguistic factors, not factorized neural weight matrices; ImplicitLanguageModel and EnergyBasedLLM describe non-autoregressive sequence energies rather than these next-word conditionals. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin
|
primary paper
full_text
|
ID: jmlr-v3-bengio03a
sha256:0bddf9a608d6…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T18:27:33Z
Updated: 2026-10-11T18:27:33Z
Source File:
data/curated/NeuralProbabilisticLanguageModel.yaml