Proximal policy optimization
Algorithms AIMech:ProximalPolicyOptimization DRAFT
Aliases: PPO
A family of policy-gradient algorithms alternating collected interaction data with repeated optimization of a surrogate objective using clipping or an adaptive KL penalty.
Identity Notes & Concept Scope
PPO is a policy-optimization algorithm family. It is distinct from the broad reinforcement-learning task and from DPO's direct preference-fitting objective; PPO is not synonymous with its clipped variant alone.
Claims & Grounded Evidence
1 Claim
Mechanism
#mechanism
The clipped variant takes the minimum of unclipped and clipped probability-ratio advantage terms; another variant uses an adaptive KL-divergence penalty, and the algorithm performs multiple minibatch epochs on collected trajectories.
⚠️ Curated Limitations & Negative Results
- Clipping the surrogate objective is not an explicit hard bound on all policy changes.
- The record covers the original clipped and KL-penalized variants without asserting task-independent performance.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ReinforcementLearning
Reinforcement Learning
|
related_match | PPO is a particular policy-gradient procedure within reinforcement-learning practice; the AIO class denotes the broader task. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
original-paper
|
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov
|
primary paper
full_text
|
ID: arXiv:1707.06347v2
sha256:e78feadadbdb…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T03:12:53Z
Updated: 2026-10-11T03:12:53Z
Source File:
data/curated/ProximalPolicyOptimization.yaml