Home / Curated / Learning Paradigms / Temporal-difference learning

Temporal-difference learning

Learning Paradigms AIMech:TemporalDifferenceLearning DRAFT

Aliases: TD learning

A family of prediction-learning methods that assign temporal credit using discrepancies between successive, recursively related predictions of future outcomes, with intervening costs or rewards included when the target is cumulative.

Identity Notes & Concept Scope This family-level record covers temporal-difference prediction and the original TD(lambda) construction, including TD(0) and TD(1) endpoints. It is not a policy optimizer, the GAE advantage estimator, backpropagation as a way to compute derivatives, a Markov-chain model, or every sequential prediction problem. TD(lambda) is treated here as a specified family member rather than a synonym for the whole paradigm. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 298 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in worktree and main, returning 20 and 20 lines respectively. All 17 unique hit contents were inspected: prior TD(lambda) and Q-routing leads, GAE and AntNet boundary statements, and their saved reports/reviews. These were references to methods rather than existing candidate records. Six exact AIO CLASS nodes with scoped synonyms/comments and outgoing edges and six native neighboring mechanisms were inspected. Absence is bounded to this snapshot and semantic neighborhood.

Claims & Grounded Evidence

2 Claims

Mechanism #method

For the original terminal-outcome problem, predictions P_t=P(x_t,w) estimate the same eventual scalar outcome z and P_(m+1)=z. With weights held fixed within a sequence, compute increments Delta w_t=alpha*(P_(t+1)−P_t)*e_t, where e_t=sum_(k=1..t) lambda^(t−k)*grad_w P_k and 0<=lambda<=1; apply their sum after the sequence. The trace can be maintained recursively as e_t=grad_w P_t+lambda*e_(t−1). TD(0) uses only the current prediction gradient, whereas TD(1) gives the same per-sequence changes as the corresponding supervised outcome-error update under the fixed-weight convention. In the cumulative-outcome extension, replace the difference with c_(t+1)+P_(t+1)−P_t and set the terminal remaining prediction to zero. This is the source’s undiscounted cumulative formulation, not an implicit insertion of a discount factor. Intra-sequence weight updates are a separate extension requiring consistent parameter timing; simple algebraic equivalence at fixed weights does not establish equality of online trajectories.

supports Source: Learning to predict by the methods of temporal differences · Locator: Sections 2.1–2.3, PDF pp.5–8 / journal pp.13–16, Equations 1–4 and trace recursion; Theorem 2 statement, PDF pp.16–17; Sections 5.1–5.3, PDF pp.25–28 / journal pp.33–36; Section 6.2, PDF pp.29–30

Scope #control-application

TD prediction can supply value estimates in a reinforcement-learning control procedure. The author exposition estimates the current behavior policy’s action values using TD updates while improving that policy toward greedy choices. Prediction alone does not specify the action-selection and policy-improvement components of a complete control algorithm.

supports Source: Reinforcement Learning: An Introduction · Locator: Section 6.4, PDF pp.155–156 / printed pp.137–138, TD prediction applied to on-policy control and Sarsa pseudocode

⚠️ Curated Limitations & Negative Results

The prediction targets must obey the declared temporal recursion. Consecutive forecasts of different future events cannot simply be subtracted; Section 5.3 embeds fixed-horizon forecasts in a larger set of related predictions. Terminal-outcome and cumulative-outcome formulations use different terminal conventions.
The original linear TD(0) theorem is convergence in expected value for an absorbing Markov setting, linearly independent state observation vectors, and sufficiently small constant step size, with weight changes after each sequence. It is not an almost-sure convergence theorem for arbitrary online nonlinear networks or a finite-data guarantee; the proof appendix was not independently audited.
TD(1)’s fixed-weight per-sequence equivalence does not make all TD methods identical to Monte Carlo or supervised learning. Updating weights within a sequence changes predictions and traces; Section 5.2 warns against mixing predictions computed with inconsistent parameter versions.
Temporal credit assignment and structural backpropagation are compatible but distinct. The method family alone supplies neither a control policy nor an exploration strategy. No claim of universal accuracy, lower computation, or benchmark superiority is curated.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ReinforcementLearning` Reinforcement Learning	related_match	TD prediction supplies value estimates for reinforcement-learning control, as directly illustrated by the author book’s Section 6.4. The local family also covers prediction without action selection. The pinned AIO target is a learning task; this connection asserts neither equivalence nor that every TD prediction problem is a control task.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`method-source`	Learning to predict by the methods of temporal differences Richard S. Sutton	primary paper full_text	ID: doi:10.1007/BF00115009 sha256:f1153204158b…
`author-book-draft`	Reinforcement Learning: An Introduction Richard S. Sutton, Andrew G. Barto	author technical report full_text	ID: http://incompleteideas.net/book/bookdraft2016sep.pdf sha256:94b195c3d5f7…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-11T14:02:32Z

Updated: 2026-10-11T14:05:14Z

Source File: data/curated/TemporalDifferenceLearning.yaml