Peng and Williams Q(lambda)

⚠️ Curated Limitations & Negative Results

• With active multi-step credit (gamma*lambda>0), exploratory rewards need not evaluate the policy implicit in the greedy maxima. The article therefore does not guarantee convergence to Q-star under arbitrary continuing exploration. Its suggestions to reduce lambda or Boltzmann temperature are not a universal convergence proof. The zero-product boundary is separately described and is not covered by a blanket positive-lambda warning.
• Equation 8 includes an additional summation caused by changing value estimates. The small-learning-rate discussion does not establish exact forward/backward equality for arbitrary online updates, a uniform finite-step residual bound or convergence of every nonlinear extension.
• The source discount domain is 0<=gamma<1, with trace parameter lambda in [0,1]. The finite recurrence remains meaningful at lambda=1, but that endpoint is not justified by literally substituting lambda=1 into the displayed (1-lambda)-weighted infinite mixture. Return convergence and any limiting interpretation require their own conditions. Repeated visits add new trace mass, so attenuation of old credit is not monotonic decay of the total accumulated trace.
• The selected primary algorithm is the tabular Figure 1 baseline. The paper discusses connectionist implementation, and Rummery’s report presents a separate neural extension with its own timing. Neither mention establishes that all neural approximators share the tabular update guarantees or that the two papers are interchangeable specifications.
• Figure 1 is a continuing do-forever loop and does not specify every episodic terminal bootstrap, trace reset or final-update convention. Sparse pruning, replacing traces, clipping and finite-precision choices alter or approximate the literal recurrence and require explicit choices. No complete runnable implementation is reconstructed here.
• The operational definition remains the 1996 journal Figure 1 method. The inspected 1994 author manuscript supports the same two-error and trace-update order, but its figure’s greedy-action line conflicts with adjoining exploration prose; the local V definition prints arg max despite an earlier max definition, and Equation 12 lacks the visible joining plus before its changing-value sum. The 1996 presentation explicitly uses an exploration policy, max and the plus sign. These are source-display observations, not a formal erratum. The 1994 cart-pole experiment uses separate current-pair and past-pair learning rates and resets traces at each trial; those experimental choices do not replace the single-rate core or specify every episodic convention.
• The reported task comparisons and claims about faster learning or the necessity of multi-step propagation are not universal results over environments, representations or computational budgets. No author software, benchmark run or original convergence proof was executed or certified by this evidence assessment.