Home / Curated / Architectures / Log-bilinear language model

Log-bilinear language model

Architectures AIMech:LogBilinearLanguageModel DRAFT

Aliases: LBL

A next-word language architecture that linearly combines position-specific transforms of context-word embeddings and normalizes their bilinear compatibility with candidate-word embeddings over the vocabulary.

Identity Notes & Concept Scope This is the original Mnih–Hinton LBL construction. It uses the same word-embedding matrix for context and candidate words, has no stochastic hidden variables, and retains a flat vocabulary normalizer. HLBL replaces that output parameterization with tree-node decisions. The FRBM and temporal FRBM in the same 2007 paper are separate latent-variable architectures, not aliases for LBL. All 442 pinned AIO labels/scoped synonyms and 355 preceding native IDs/names/aliases were compared, with no normalized identity match. Full-checkout fixed-string searches included ignored and hidden files in both worktree and main, returning 54 lines each. All 41 unique matched contents were read: source references, complete-model contrasts, frontier leads and prior review context for the hierarchical-softmax component. None is an existing record for these complete architectures. Twenty exact AIO nodes and their generated records plus four complete native neighbors were inspected. Absence is bounded to this snapshot and semantic neighborhood.

Claims & Grounded Evidence

1 Claim

Mechanism #method

In the 2007 row-vector convention, let r_w be a word’s embedding written as a column, C_i the transform for context position i, b_r a feature bias and b_w a candidate-word bias. The candidate score is sum_i r_context_i^T C_i r_w+b_r^T r_w+b_w. Equivalently, form r_hat=b_r+sum_i C_i^T r_context_i once, then take score_w=r_hat^T r_w+b_w and apply the vocabulary softmax. The original energy is the negative of this score. Maximum-likelihood gradients are observed sufficient statistics minus their exact finite-vocabulary conditional expectations. Because the embedding matrix is shared, its gradient includes both input-context and candidate-output roles. The 2008 exposition writes r_hat=sum_i C_i r_context_i with the opposite matrix orientation and without an explicit feature bias; this is a notation/parameterization distinction to preserve when comparing formulas.

supports Source: Three New Graphical Models for Statistical Language Modelling · Locator: Section 4 and 4.1, Equations (13)–(15), printed pp.645–646 / PDF pp.5–6; feature and bias conventions, Equation (2), printed p.642 / PDF p.2
supports Source: A Scalable Hierarchical Distributed Language Model · Locator: Section 3, Equations (1)–(2), PDF/printed pp.2–3; shared embedding and linear hidden-feature discussion, p.3

⚠️ Curated Limitations & Negative Results

The context feature computation is linear, while normalization makes word probabilities nonlinear. Position-specific transforms preserve context order; the construction is not generally a bag-of-words model.
Without an additional exact computational method, the original flat normalizer requires summing candidate scores over the vocabulary. Absence of stochastic hidden variables makes this expectation explicit, not independent of vocabulary size.
Bilinear scores and shared learned embeddings do not make joint parameter fitting a convex optimization problem. Exact likelihood gradients do not guarantee a global optimum or calibrated predictions.
The energy scores one candidate conditional on its context; it is not the unnormalized energy of a whole sequence used by the pinned ImplicitLanguageModel or EnergyBasedLLM classes.
The record covers a fixed-context word model, without asserting long-context capability, benchmark superiority, author-code behavior or the mechanisms of the neighboring FRBM variants.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:AutoregressiveLanguageModel` Autoregressive Language Model	broad_match	The source specifies next-word conditionals given preceding words, defining a sequential language model with a finite context. The AIO target is broader than this position-specific bilinear context and flat vocabulary-output architecture. The mapping uses its sequential-probability scope, not its qualitative assertion about excelling at generation. HierarchicalLanguageModel denotes multilevel representations and is not an exact identity for a vocabulary-tree construction; OutputLayer, SoftmaxLayer and EmbeddingLayer are components rather than whole-model superclasses.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`method-source`	Three New Graphical Models for Statistical Language Modelling Andriy Mnih, Geoffrey Hinton	primary paper full_text	ID: 10.1145/1273496.1273577 sha256:d0709a403248…
`later-exposition`	A Scalable Hierarchical Distributed Language Model Andriy Mnih, Geoffrey Hinton	primary paper full_text	ID: https://proceedings.neurips.cc/paper/2008/hash/1e056d2b0ebd5c878c550da6ac5d3724-Abstract.html sha256:1532535af39d…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-11T18:15:42Z

Updated: 2026-10-11T18:15:42Z

Source File: data/curated/LogBilinearLanguageModel.yaml