Direct preference optimization
Algorithms AIMech:DirectPreferenceOptimization DRAFT
Aliases: DPO
Preference-based policy fitting using a classification objective on policy-to-reference probability ratios for preferred and dispreferred responses.
Identity Notes & Concept Scope
A training algorithm, not an instruction-tuned language-model instance or a synonym for all reinforcement learning from human feedback.
Claims & Grounded Evidence
1 Claim
Mechanism
#mechanism
A reward-to-policy reparameterization yields a logistic preference loss that trains the policy without separately fitting an explicit reward model.
⚠️ Curated Limitations & Negative Results
- The derivation uses a specified preference model and KL-regularized reference-policy objective; it is not a guarantee of universally correct human preferences.
- No benchmark superiority or general safety claim is curated.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:TrainingStrategy
TrainingStrategy
|
related_match | DPO is a policy-fitting procedure. The target discusses training strategies but uses a preprocessing genus, so a subclass claim is withheld. |
AIO:ReinforcementLearningLLM
Reinforcement Learning LLM
|
related_match | DPO is motivated by preference-based language-model alignment but its derived training objective does not require the separate RL optimization loop described for conventional RLHF. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
original-paper
|
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn
|
primary paper
full_text
|
ID: arXiv:2305.18290v3
sha256:92cb3a2b7136…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T03:02:19Z
Updated: 2026-10-11T03:02:19Z
Source File:
data/curated/DirectPreferenceOptimization.yaml