REINFORCE
Algorithms AIMech:REINFORCE DRAFT
Aliases: REINFORCE algorithm
A family of stochastic reinforcement-learning update rules that multiply a sampled reward relative to an admissible baseline by the derivative of the log probability of the sampled action or unit output, with a nonnegative learning-rate factor.
Identity Notes & Concept Scope
This record covers Williams’s REINFORCE update family, including its basic immediate-reward rule and the stated fixed-episode extension. The characteristic eligibility is a score-function derivative. This is not the entire reinforcement-learning task, an action-value bootstrapping rule, PPO’s clipped surrogate, a generic optimizer, or every estimator that uses random perturbations. The acronym names the update structure; no trained policy instance is represented. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 334 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in worktree and main, returning 66 and 66 lines respectively. Match-centered excerpts from all 41 unique hit contents were inspected: ordinary verb reinforce in existing ant/search/pseudo-label records and calculations, plus neighboring Gumbel straight-through discussions and the new gradient-estimator discovery lead, not existing candidate records. Exact checkout counts and differences are retained in the ledger. Ten exact AIO CLASS nodes with scoped synonyms/comments and outgoing edges and six full native neighboring records were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
3 Claims
Mechanism
#method
For a stochastic unit with conditional mass or density g_i(y_i;w_i,x_i), sample its output y_i, receive scalar reinforcement r and update weight w_ij by alpha_ij*(r-b_ij)*d log(g_i)/d w_ij. The baseline b_ij is conditionally independent of the current unit output given the weights and unit inputs; it may depend on permitted input context or past experience. In the original immediate-reward formulation, the nonnegative rate factor may depend at most on the unit’s parameter vector and trial index; it is not chosen from the current sampled output. Under the stated differentiability and environmental assumptions, the baseline-centered score product estimates the corresponding expected-reward gradient component. A common scalar rate gives an expected update proportional to that gradient; differing nonnegative coordinate rates generally rescale its components and need not preserve collinearity. For a Bernoulli-logistic unit with probability p and input x_j, the eligibility is (y-p)*x_j. In the fixed-length episodic extension, hold parameters fixed during the episode, sum each parameter’s score eligibilities over time, and multiply that sum by terminal reward minus an admissible episode baseline when updating at the end.
Scope
#baseline-timing
Williams’s reinforcement-comparison example computes the reward baseline from past experience and explicitly excludes the current output or current reward from its computation. The expected-update theorem distinguishes a common scalar step size from nonnegative coordinate-dependent rates.
Limitation
#baseline-variance
For the Bernoulli-logistic score h-p, Bengio and colleagues derive the variance-minimizing constant baseline as the score-squared-weighted mean of the loss, rather than its ordinary mean. This is a minimum within the specified constant-baseline family, not among all possible gradient estimators.
⚠️ Curated Limitations & Negative Results
- The score identity requires valid differentiable probabilities or densities and conditions permitting differentiation of the expectation. Parameter-dependent support or extra direct reward dependence requires additional analysis; a discrete black-box reward need not itself have a derivative with respect to the action.
- A baseline that depends on the same sampled action or its current reward does not automatically preserve the unbiased score identity. An input-dependent baseline or a statistic based on past trials can satisfy the stated conditional-independence requirement.
- Baseline choice changes variance and can increase it. The mean reward is not generally the minimum-variance baseline, and low variance is not guaranteed simply by naming a baseline.
- Expected gradient alignment is not monotonic improvement of each sampled update, a global optimum guarantee, or a convergence theorem for arbitrary step sizes. Zero coordinate rates can prevent movement even at a nonzero gradient.
- The original episodic construction assumes a declared finite episode and parameters fixed throughout it. It is not a blanket justification for arbitrary online bootstrapping, truncation, discounting, off-policy sample reuse or action-dependent baselines.
- The primary journal paper was read through a public third-party copy and checked against publisher metadata. Only the declared method pages were inspected; its full proof appendix and empirical literature were not independently audited.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ReinforcementLearning
Reinforcement Learning
|
related_match | The primary source explicitly develops a stochastic reinforcement-learning algorithm family driven by reward feedback. The exact AIO target denotes the broader learning task. This is an algorithm-to-task related match, not equivalence or a claim that every reinforcement learner follows this gradient rule. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Ronald J. Williams
|
primary paper
full_text
|
ID: doi:10.1007/BF00992696
sha256:e1029df5459f…
|
bernoulli-baseline-source
|
Yoshua Bengio, Nicholas Léonard, Aaron Courville
|
primary paper
full_text
|
ID: arXiv:1308.3432v1
sha256:509eb5e5294d…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T16:43:04Z
Updated: 2026-10-11T16:43:04Z
Source File:
data/curated/REINFORCE.yaml