Document NADE
Architectures AIMech:DocumentNADE DRAFT
Aliases: DocNADE
A neural autoregressive document model that predicts successive words from shared additive embeddings of the preceding words, using normalized word conditionals and a deterministic document representation.
Identity Notes & Concept Scope
This is Larochelle and Lauly’s original Document NADE. The final construction uses binary-tree word probabilities, with a flat-softmax intermediate derivation. It is inspired by Replicated Softmax but is a separately normalized directed model with deterministic hidden features, not the same undirected latent-variable model. Its full-document representation is invariant to word permutation, while its unsymmetrized sequence probability need not be. Supervised and distributional-prior extensions require separate scope decisions. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 349 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in worktree and main, returning 5 and 5 lines respectively. All 3 unique matched line contents were inspected in full: the prior NADE-variant lead, RNADE source URL and previous review exclusion, not existing candidate records. Exact checkout counts and differences are retained in the ledger. Seventeen exact AIO CLASS nodes with scoped synonyms/comments and outgoing edges and five full native neighboring records were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
For a document of given length D, form h_i=sigmoid(c+sum_(k<i) W[:,v_k]); the same word embedding and output parameters are reused across positions, and the preactivation is updated by adding the newly observed word’s embedding. A flat softmax can normalize the next-word distribution. The paper’s final model replaces it with a binary tree whose leaves are vocabulary words: each internal node predicts the probability of taking its right branch, and a word probability is the product of its observed left/right branch probabilities. Multiply all observed-word conditionals to obtain a normalized probability over length-D sequences. Train by differentiating their negative log likelihood. After all words, h(v)=sigmoid(c+sum_i W[:,v_i]) is an order-invariant document representation; tanh is also tested. For count-vector input, the paper samples word occurrences without replacement to produce uniformly random distinct orderings and trains on those sequences. Its separate symmetrized sequence model averages probabilities over distinct permutations; training on sampled order log losses is a surrogate for that log mixture. A count-vector probability instead sums the unsymmetrized sequence probabilities over its distinct permutations.
⚠️ Curated Limitations & Negative Results
- Order invariance of the summed document representation does not imply exchangeability of the sequence likelihood. The symmetrized probability of each sequence differs from its count-vector probability by the number of distinct permutations; duplicate words change that number.
- For a word whose tree path has length L, conditional output work is O(HL). The claimed logarithmic vocabulary scaling requires suitable tree depth or average depth; a full binary tree alone can have linear depth. The record does not assert a logarithmic worst case for every full tree.
- The main paper’s text below Equation (11) retains vi=1 in the joint product although words are categorical. The curated likelihood uses the observed word and its branch path, consistent with Equations (9) and (11) and the supplement’s probability computation.
- For Bernoulli tree training, the negative-log-likelihood derivative is the probability of the right branch minus the observed branch indicator. The supplement’s p(actual branch)-indicator notation must not be read literally as the probability of the observed left branch when that indicator is zero. No author implementation behavior is inferred.
- The stated model is conditional on document length and does not supply a learned document-length distribution. Shared additive prefix features discard order within the observed prefix. Hidden topic features are deterministic representations, not posterior latent-topic probabilities.
- No classification/retrieval advantage, semantic quality guarantee, exact symmetrized-mixture training gradient or implementation certification is asserted.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:BayesianNetwork
Bayesian Network
|
broad_match | For a given length, the observed-word variables have ordered directed conditional factors, giving a fully visible Bayesian network. The AIO target is broader than this shared additive-embedding and tree-output architecture. Deterministic topic features do not add stochastic latent nodes. |
AIO:ProbabilisticTopicModel
Probabilistic Topic Model
|
related_match | The source explicitly develops a probabilistic document/topic representation model, matching the topical-analysis context. The pinned AIO target lies in a task hierarchy and mentions themes and temporal changes, whereas this native identity specifies one architecture without requiring temporal modeling. A related match preserves that abstraction difference without forcing an architecture-to-task subclass or equating hidden features with normalized topic distributions. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Hugo Larochelle, Stanislas Lauly
|
primary paper
full_text
|
ID: https://proceedings.neurips.cc/paper/2012/hash/b495ce63ede0f4efc9eec62cb947c162-Abstract.html
sha256:7811d256a50c…
|
supplement-source
|
Hugo Larochelle, Stanislas Lauly
|
primary paper
full_text
|
ID: https://proceedings.neurips.cc/paper/2012/hash/b495ce63ede0f4efc9eec62cb947c162-Abstract.html#supplement
sha256:fa234cf5c169…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T17:49:48Z
Updated: 2026-10-11T17:49:48Z
Source File:
data/curated/DocumentNADE.yaml