WIS-TO-GTD(lambda)

⚠️ Curated Limitations & Negative Results

• The selected explicit implementation is the author WTOGTD class at commit d73d441ca34c546cae79c9289104a68238c9b7ce. Its own-history accumulating correction trace differs from the old-main-trace operand printed in the 2015 Algorithm 3. The code supports a version-specific interpretation, not a formal erratum or blanket paper/code equivalence. It remains distinct from WIS-TD with provisional state, accumulating WIS-GTD, ordinary true-online GTD and the on-policy usage-based hosts.
• The paper reciprocal at zero is zero; the code uses one at an exactly zero usage denominator. Bare eta>=0 does not guarantee positive or bounded usage and reciprocal steps for arbitrary features and ratios. For example, scalar u_0=1, eta=2 and phi_0=rho_0=1 give u_1=0. With zero initial weights and traces, R_1=1 and gamma_1=0, the paper convention gives theta_1=0 while the code convention gives theta_1=1. This hand boundary example lies outside sufficiently small eta positivity conditions; it is not an observed experimental failure or a convergence result.
• The pinned constructor initializes theta, w, all three traces, v, previous rho and previous prediction to zero, with uniform configured initial usage. Its initepisode method clears only the main trace z. Usage and auxiliary/correction histories are not thereby fully reset. A contiguous continuing trajectory and an episodic caller that resets only z need not have the same state carry; no universal full-reset protocol is inferred from the printed algorithm.
• Meaningful finite importance ratios require suitable target-policy action support under behavior; relevant features, rewards and steps must also be well defined. The implementation does not enforce all such conditions. Main alpha is componentwise and auxiliary beta remains scalar. With beta=0 and zero initial w, the gradient correction vanishes and cannot distinguish competing correction-trace recurrences; beta=0 alone is insufficient for arbitrary nonzero w.
• Old v, old w, pre-update theta and the previous pre-update next-state prediction are part of the stated timing. Substituting new w in the main correction or updating the stored prediction from new theta changes that procedure. A next lambda of one removes the explicit gradient-correction term but does not alone make the method an exact Monte Carlo average or normalized WIS estimator.
• The source leaves convergence analysis of the new WIS/usage algorithms for future work. The 2014 host’s exact linear sampled forward-view construction and the 2015 usage correspondence do not establish a general convergence, stability, monotonic improvement or exact per-transition objective-gradient theorem for this implementation. The separate WIS-TD forward-weight theorem is not silently transferred to WIS-TO-GTD.
• The scope is fixed linear state-value prediction. Nonlinear or learned features, action-value control and policy improvement require separate evidence. The record makes no experimental performance or author-test-pass claim; reading source code and configuration files is not executing or reproducing them.
• The selected paper evidence is the acquired eighteen-page combined author copy of the 2015 paper and Appendix A plus the specific first-author-hosted 2014 GTD version. Native-text version comparisons and the embedded PDF creation date do not certify every formula or establish a scientific revision date. The commit-pinned implementation scope does not certify all repository callers, dependencies, later versions or proofs.