Home / Curated / Architectures / RNN-NADE

RNN-NADE

Architectures AIMech:RecurrentNeuralNetworkNeuralAutoregressiveDistributionEstimator DRAFT

Aliases: Recurrent neural network neural autoregressive distribution estimator

A neural sequence architecture in which a deterministic recurrent state supplies parameters of a NADE output distribution, combining temporal recurrence with normalized within-frame autoregressive conditionals.

Identity Notes & Concept Scope This is the RNN-NADE identified in the 2012 recurrent-output-estimator paper by substituting NADE for the conditional RBM. Temporal order between vector frames and variable order within a frame are separate. The recurrent state and NADE prefix activations are deterministic computations; there is no stochastic RBM hidden state to integrate out. It is distinct from unconditional NADE, order-agnostic NADE, RNN-RBM and a recurrent predictor with independent Bernoulli outputs. All 442 pinned AIO labels/scoped synonyms and 364 preceding native IDs/names/aliases were compared, with no normalized identity match. Full-checkout fixed-string searches included ignored and hidden files in both worktree and main, returning 129 lines each. All 91 unique matched contents were read, including 58 initial contents and 33 additional saved-review contents: prior model contrasts, evidence references and discovery leads, rather than existing candidate identities. All 52 semantic label/synonym hits and 22 exact annotated AIO nodes were inspected, alongside corresponding generated records and four complete native neighbors. Previously read contextual files were checked for byte identity. Absence is bounded to this snapshot and inspected neighborhood.

Claims & Grounded Evidence

1 Claim

Mechanism #method

For the single-layer binary specialization, compute the past-dependent biases b_t=b+P r_(t-1) and c_t=c+Q r_(t-1). Fix an ordering of D bits within the current frame. At position i, compute z_(t,i)=sigmoid(c_t+W[:,<i] v_(t,<i)) and q_(t,i)=sigmoid(b_(t,i)+V[i,:] z_(t,i)). Multiply q_(t,i) raised to v_(t,i) times (1-q_(t,i)) raised to (1-v_(t,i)) over positions, then multiply frame probabilities over time. Reuse a prefix accumulator initialized to c_t, adding W[:,i] v_(t,i) after evaluating each conditional. After the frame is observed or generated, update the separate recurrent state using its own input and recurrent weights. Differentiate the exact sum of frame negative log probabilities through the NADE parameters, the bias projections and the recurrent history. The 2012 paper explicitly contrasts these exact model gradients with CD-based RNN-RBM gradients. Ancestral generation samples bits in the fixed within-frame order before updating the recurrent state.

supports Source: Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription · Locator: NADE substitution in Section 2, PDF pp.2–3; RNN architecture and explicit RNN-NADE exact-gradient statement, Section 4 and Figure 2, pp.3–4; temporal BPTT and tractable-model discussion, Sections 4.2 and 6, pp.5–6
supports Source: The Neural Autoregressive Distribution Estimator · Locator: Sections 3–5, Equations (3), (9)–(11), Figure 1 and Algorithm 1, printed pp.30–33 / PDF pp.2–5; probability and accumulator notation qualified against chain-rule definition

⚠️ Curated Limitations & Negative Results

Exact normalized probabilities and gradients apply to this specified model with fully observed training sequences. They do not imply global optimization, calibration, equality to an RBM distribution or tractable inference for every pattern of missing frames or bits.
The within-frame ordering and temporal recurrence serve different purposes. Prefix sharing gives O(DH) NADE evaluation per frame for H hidden units, in addition to the recurrent-state and bias-projection costs; it does not remove sequential generation dependencies.
The original NADE paper prints an accumulator index discrepancy and initializes a multiplicatively accumulated probability to zero with an incorrect plus between Bernoulli factors in Algorithm 1. The displayed mechanism follows its normalized chain definition, Equations (9)–(11) and prefix-update rule, retaining the source qualification.
Output weights V may be learned separately from W as in the original NADE paper. Sharing prefix parameters across conditionals does not require tying input and output weights, and it does not make within-frame variables independent.
The RNN-RBM paper’s negative joint/bias signs concern its RBM formulas and are not a prescription for NADE conditionals. Backpropagation through the separate recurrence still requires dimensionally correct transposes. No author-code or empirical superiority claim is certified.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:RecurrentNeuralNetwork` Recurrent Neural Network	broad_match	The pinned AIO target is broader than this explicit recurrent neural architecture with a specified conditional-density output. The deterministic state has directed temporal connections; the output estimator adds a narrower structural restriction. A temporal probability model over vector frames is not necessarily an autoregressive language model.
`AIO:BayesianNetwork` Bayesian Network	broad_match	The visible variables can be ordered first by frame time and then by within-frame position, with normalized neural conditional factors depending only on earlier visible variables. This is a fully visible acyclic probabilistic dependence model, so the pinned BayesianNetwork is broader. Deterministic recurrent and prefix computations parameterize factors and do not imply intervention-level causality.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`method-source`	Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription Nicolas Boulanger-Lewandowski, Yoshua Bengio, Pascal Vincent	primary paper full_text	ID: arXiv:1206.6392 sha256:7660a28d9292…
`nade-component`	The Neural Autoregressive Distribution Estimator Hugo Larochelle, Iain Murray	primary paper full_text	ID: pmlr-v15-larochelle11a sha256:b1fe502ea1b3…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-11T18:53:57Z

Updated: 2026-10-11T18:53:57Z

Source File: data/curated/RecurrentNeuralNetworkNeuralAutoregressiveDistributionEstimator.yaml