Home / Curated / Architectures / Retentive network

Retentive network

Architectures AIMech:RetNet DRAFT

Aliases: RetNet

An autoregressive neural sequence architecture that stacks pre-normalized residual blocks combining gated multi-scale retention with feedforward transformations, with parallel, recurrent and chunkwise forms of retention.

Identity Notes & Concept Scope RetNet denotes Sun and colleagues' architecture, not a generic recurrent network or an attention-based Transformer language model. Its retention operation is a component, while the architecture also specifies gating, normalization, residual paths and feedforward blocks.

Claims & Grounded Evidence

1 Claim

Mechanism #sequence-mechanism

Retention projects position-modulated queries, keys and values and combines causal query-key interactions with exponential distance decay. Different heads use fixed decay scales; headwise normalization and a swish gate combine their outputs. Recurrent evaluation updates a decayed key-value state, and chunkwise evaluation combines parallel within-chunk interactions with recurrent cross-chunk state.

supports Source: Retentive Network: A Successor to Transformer for Large Language Models · Locator: Sections 2.1–2.3, equations 5–9 and Figures 3–4, PDF/printed pp. 3–5

⚠️ Curated Limitations & Negative Results

Constant recurrent decoding cost is with respect to prior sequence length at fixed model dimensions, not constant total work for a whole sequence or arbitrary model size.
The mathematical parallel, recurrent and chunkwise representations do not by themselves establish identical finite-precision outputs for every implementation; numerical score-normalization details matter.
This record does not generalize the paper's reported benchmark or hardware results.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	broad_match	Learned retention projections, gates and feedforward transformations compose a neural architecture. ArtificialNeuralNetwork is broader and does not incorrectly identify retention as Transformer attention.

Evidence Sources & Primary Literature

1 Source

Source ID	Publication / Document	Type / Access	Reference & Checksum
`original-paper`	Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei	primary paper full_text	ID: arXiv:2307.08621v4 sha256:b75fdb67fdcc…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-11T04:26:45Z

Updated: 2026-10-11T04:26:45Z

Source File: data/curated/RetNet.yaml