Home / Curated / Algorithms / Modified Connectionist Q-Learning

Modified Connectionist Q-Learning

Algorithms AIMech:ModifiedConnectionistQLearning DRAFT

The original neural action-value learning procedure that uses a selected-next-action temporal-difference error to update weights through accumulated selected-output gradient eligibility traces, retaining credit through exploratory actions in its stated online sequence.

Identity Notes & Concept Scope The selected identity is Rummery–Niranjan neural MCQ-L with the actual selected-next-action error and accumulating parameter-gradient traces. Maintained one-step Sarsa expressly distinguishes that complete neural method; shared lambda-zero error does not establish whole-method identity. QLearning and the original TD prediction family remain separate. Empty aliases avoid unresolved Sarsa(lambda) naming equivalence and chronology. Fresh collection at actual merged base 2fea41895243fec301f372eb2a2e1247cf76e708 on 15 October 2026, 13:57:03–13:57:48 UTC, compared all 760 native and 442 pinned AIO records: 4,798 ID/local-ID/name/alias/scoped-synonym occurrences and 3,667 normalized values, with zero candidate collisions and 17 contextual fields. Four checkout searches included ignored and hidden files and returned 2,858, 2,876, 7,119 and 8,022 occurrences (20,875 total), retained as 880 distinct matching contents. Root read the 19 focused candidate-content projections, all 17 normalized context fields, all 13 matched frontier objects, ten neighboring identity/definition/mechanism projections and fourteen exact AIO CLASS nodes with 59 incident-edge entries. This is bounded semantic inspection; root does not claim a verbatim reading of every broad context-only occurrence or all 880 groups. All 909 main and 393 nested objects were queried recursively for candidate/context fields. Historical primary and draft inspections retain their separately recorded source scopes.

Claims & Grounded Evidence

6 Claims

Mechanism #selected-next-action-error

For an ongoing transition from selected state-action pair (x_t,a_t) to (x_(t+1),a_(t+1)), use the scalar temporal-difference error delta_t = r_t + gamma*Q(x_(t+1),a_(t+1)) - Q(x_t,a_t). The next action is the action selected by the behavior policy, rather than necessarily a maximizing action. An exploratory action does not by itself require the eligibility trace to be cut.

supports Source: On-line Q-learning using connectionist systems · Locator: Section 3.2, PDF pp. 7–8 / printed pp. 6–7, Equation 9 and the selected-action versus maximum discussion.

Mechanism #accumulating-neural-gradient-trace

Maintain one eligibility value for each neural weight. If g_t is the selected-output gradient computed at step t, accumulate e_t = g_t + gamma*lambda*e_(t-1), starting from zero eligibility. An ongoing selected-action prediction error scales the retained eligibility vector in the weight update. Earlier gradient contributions are those stored from their respective computation times, not a requirement to recompute the whole trajectory at one later parameter vector.

supports Source: On-line Q-learning using connectionist systems · Locator: Section 4, PDF pp. 8–10 / printed pp. 7–9, Equation 13 recursive expression, Figure 1 and its following explanation; Equation 9 on PDF p. 7.

Mechanism #printed-online-operation-order

Figure 1 resets eligibilities at trial initialization and, on an ongoing iteration, selects an action before updating weights from the preceding reward, the selected current value, the cached preceding value and the preceding trace. It then computes the selected-output gradient, updates the trace, executes the selected action and receives its reward. The description preserves that printed operation order rather than substituting a later trace algorithm.

supports Source: On-line Q-learning using connectionist systems · Locator: Section 4, PDF p. 9 / printed p. 8, Figure 1 steps 1–8; explanatory paragraph on PDF p. 10 / printed p. 9.

Mechanism #single-trace-and-inactive-action-networks

A single selected-action error permits one eligibility trace per weight. When each action has a separate single-output network, weights in nonselected action networks receive no new selected-output gradient, while their old traces are multiplied by gamma*lambda. These old traces decay when gamma*lambda<1 and are retained when gamma*lambda=1. With shared weights, this does not imply that only one action output can change.

supports Source: On-line Q-learning using connectionist systems · Locator: Section 4, PDF pp. 9–10 / printed pp. 8–9, paragraph following Figure 2 and Equation 14; discount and trace domains in Sections 2.2 and 3, PDF pp. 4–5.

Scope #zero-trace-and-greedy-special-cases

When lambda=0, the accumulating trace contains only the selected-output gradient from the current step. Under matching value, representation and update conventions, a greedy next action makes the selected-next-action value equal to the next-state maximum. These one-step or transition-level coincidences do not identify the complete neural trace procedure with every tabular Sarsa or Q-learning algorithm.

supports Source: On-line Q-learning using connectionist systems · Locator: Section 3.2, PDF pp. 7–8 / printed pp. 6–7, Equation 9 and greedy-action discussion; Section 6.1, PDF p. 17 / printed p. 16.

Scope #neural-control-method-scope

The inspected MCQ-L method combines neural action-value approximation, scalar reward feedback and online accumulating eligibility credit. The report studies robot-control realizations and exploratory action-selection settings, but the particular robot, network size and Boltzmann temperature schedule are experimental choices rather than a universal implementation specification.

supports Source: On-line Q-learning using connectionist systems · Locator: Sections 2.2–2.3, PDF pp. 4–5 / printed pp. 3–4; Sections 3.2–5, PDF pp. 7–11 / printed pp. 6–10; Section 6.1, PDF p. 17 / printed p. 16.

⚠️ Curated Limitations & Negative Results

Equation 13 literally places Q_(t+k) in a sum over k=0,...,t, whereas Equation 9 uses past Q_k and the recursive right-hand expression uses the current gradient plus gamma*lambda times the preceding trace. The description follows the mutually consistent past-gradient mechanism and displayed recursion while retaining this source indexing discrepancy; it does not claim an author erratum or a defect in unseen software.
Figure 1’s caption calls the weight update stage 5, although the displayed update is step 4. Its steps 7–8 receive a reward and stop when the trial ends without explicitly specifying a final terminal update. The ongoing-transition description does not silently add a terminal flush, bootstrap value or reset policy and attribute it to the literal pseudocode.
Accumulating gradients computed at changing online weights does not by itself establish an exact fixed-parameter forward-return identity. The stated sequence is not a claim of true-online, Dutch-trace or replacing-trace equivalence. In particular, lambda=1 must not be justified by literal substitution into a zero-times-infinite geometric mixture of truncated returns.
The report permits gamma and lambda in [0,1]. Inactive traces are geometrically attenuated only when gamma*lambda<1; at product 1, even bounded repeated gradients can accumulate without a uniform trace bound. Gamma=1 also does not ensure that a continuing infinite-horizon reward sum exists; episodic or other summability conditions must be stated.
The source’s convergence language is not a general guarantee for nonlinear approximation, changing exploration, arbitrary learning rates or every task. Persistent exploration changes the selected-action target, and a positive temperature floor does not prove eventual greedy control. The report itself later acknowledges the missing general convergence proof and sensitivity to learning settings.
The acquired Cambridge report bears a September 1994 cover and already mentions Sutton’s proposed SARSA name in a footnote. This establishes the inspected text, not an independently resolved naming chronology, first-use priority or exact alias between the entire original neural MCQ-L method and all later Sarsa(lambda) variants.
No author implementation or historical experiment was executed for this evidence assessment. The robot-control comparisons do not establish universal speed, robustness, optimality or a preferred trace/temperature setting. Native PDF extraction was corrupted by font encoding; OCR was used for prose and the critical equations and pseudocode were inspected visually.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ReinforcementLearning` Reinforcement Learning	related_match	The method learns action values from actions, rewards and transitions for control. The pinned AIO target is a learning task; this qualified algorithm-to-task relation is neither exact identity nor a subclass assertion, a trained model instance or a transferable convergence guarantee.

Evidence Sources & Primary Literature

1 Source

Source ID	Publication / Document	Type / Access	Reference & Checksum
`rummery1994`	On-line Q-learning using connectionist systems G. A. Rummery, M. Niranjan	primary paper full_text	ID: CUED/F-INFENG/TR 166 sha256:c9434f056d17…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-15T14:06:22Z

Updated: 2026-10-15T14:06:22Z

Source File: data/curated/ModifiedConnectionistQLearning.yaml