Stein Variational Policy Gradient

⚠️ Curated Limitations & Negative Results

• Parameter-distribution entropy and state-conditional action entropy act on different random variables. Shared Stein updates do not make this identical to Soft Q-learning or amortized SVGD.
• The posterior interpretation requires a finite positive normalizer. The reported flat improper prior on an unbounded parameter space does not establish that condition. The experiments should not be described as exact posterior sampling.
• The finite population uses estimated return gradients and finite steps. The paper does not supply a general finite-particle convergence, exact-sampling, monotonically decreasing-KL or global-optimality guarantee for this practical algorithm.
• Eq9 divides by alpha, so alpha=0 is not a directly defined substitution. A separately rescaled zero-temperature limit still combines policy gradients through the kernel. Approximate independent-policy updates additionally require negligible off-diagonal weighted-gradient contributions; with the reported median-adaptive RBF bandwidth, uniformly increasing all particle separations leaves kernel values unchanged and does not by itself establish independence.
• Kernel choice, high-dimensional parameter geometry and bandwidth are consequential. The median-distance RBF rule is empirical; its bandwidth becomes zero for a zero median unless an additional safeguard is supplied. Coincident-particle degeneracy and a universally effective kernel are not resolved here. Euclidean repulsion follows from the stated RBF derivative with a fixed chosen positive bandwidth, not from positive definiteness alone.
• Parameter separation need not imply useful behavioral diversity. The state-visitation comparison is a t-SNE visualization of four selected policies and explicitly admits projection artifacts.
• Equal transition counts do not establish equal wall-clock cost. SVPG and the independent baseline select the best-return agent; the joint baseline trains a single policy.
• Table1 does not support uniform superiority on its episode column: A2C Double Pendulum lists46 for Joint and199 for SVPG. Thresholds are relative to each policy’s own best return, and the wording for negative returns is insufficient to infer a shared performance target.
• The empirical scope is four rllab continuous-control tasks, five reruns and the specified policy architecture, gradient estimators, particle count and hyperparameters; no broad RL or universal performance claim follows.
• The source versions disagree on log-standard-deviation parameterization: arXivv1 says a vector, while the official UAI version says a linear function. Code was not inspected to resolve the difference.
• The rescaled exploration/exploitation expression in UAI Section5.4 multiplies Eq9’s flat-prior direction by alpha and omits the averaging factor; it illustrates relative weights, not a numerically identical fixed-step update.
• Natural-policy-gradient, TRPO variants, adaptive temperature schedules and layer-wise kernels are proposed future directions, not demonstrated implementations or guarantees in this paper.