Watkins hard-cut multistep Q-learning

⚠️ Curated Limitations & Negative Results

• The selected scope is the original hard-cut forward-return and delayed-update construction, not every method called Q(lambda), the whole Q-learning family, selected-next-action MCQ-L/Sarsa, Peng–Williams two-error propagation, or a replacing-trace component. The thesis separately proposes a soft exponential continuation rule. That alternative is not a defining component or an exact alias of this hard-cut method.
• The inspected return-estimation development assumes bounded rewards and discount gamma<1 in a Markov setting. Continuing undiscounted returns, unbounded rewards or arbitrary approximation require additional conditions. Lambda_star=1 does not remove non-policy cuts and does not itself establish existence of a return, stable learning or convergence.
• The displayed recursion on printed p. 94 / PDF p. 101 uses U_t(s_(t+1)), while the immediately following explanatory prose invokes U_(t+1)(s_(t+1)). The draft preserves the displayed formula and records this timing gap instead of silently selecting a modern convention. Return construction, delayed updates and changing value estimates must use an explicitly declared timing convention in an implementation.
• The initial corrigenda amend the prediction-difference development on printed pp. 90–91, retaining changing-value correction terms. The original printed p. 91 update also uses C(x,t+1) after defining C(x,t). These passages do not establish exact equality between arbitrary finite-step online updates and the delayed forward construction. Small-step remarks are not a uniform error bound or a true-online equivalence theorem.
• The preliminary policy-classification discussion assumes a unique policy action per state, while the maximum-based policy can have ties. Tie breaking, the classification of equal-valued alternatives, initialization, terminal treatment and episode resets are not fully specified by this selected construction. An exploratory sampling branch can still select a maximizing action; its label alone does not determine whether the selected action is non-policy.
• The thesis explicitly does not prove convergence of this multistep control family and discusses possible interactions between changing values and policies. Its one-step result is not a general proof for this method, nonlinear approximators, arbitrary behavior or arbitrary learning rates. This is a boundary of the inspected original source, not a claim that no later theorem exists.
• Evidence is a 241-page Cambridge-hosted scan of the May 1989 thesis, including its initial corrigenda. The source’s PDF build date and repository modification date are not publication dates. Selected methods, qualifications and formula images were inspected; no full-thesis proof certification, author-code inspection or experiment reproduction is claimed. Historical demonstrations do not establish universal speed, variance reduction or benchmark superiority.