Hungry Hungry Hippos layer
Layers & Modules AIMech:H3Layer DRAFT
Aliases: H3
A sequence layer that combines projected queries, keys and values with shift and diagonal state-space models, using multiplicative interactions to associate and retrieve sequence information.
Identity Notes & Concept Scope
H3 is the layer defined in Fu and colleagues' Algorithm 1. Networks interleave H3 with MLP layers; the paper's hybrid H3-attention model additionally uses attention layers. Those complete networks and the FlashConv execution algorithm are not identical to this component.
Claims & Grounded Evidence
1 Claim
Mechanism
#sequence-mechanism
Project the input to Q, K and V; pass K through a shift state-space model. Within each head, feed the outer product of shifted keys and values through a diagonal state-space model, multiply by Q, then concatenate heads and apply an output projection.
⚠️ Curated Limitations & Negative Results
- The scalar-head elementwise formula is a special case; general heads use outer products and matrix-vector contractions.
- The complete H3 language model and hybrid attention architecture require additional blocks beyond this layer. No universal language-model performance or recall guarantee is asserted.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:Layer
Layer
|
broad_match | The defining algorithm takes a sequence representation and outputs a transformed representation for subsequent layers. AIO Layer is broader than this specific shift/diagonal state-space construction. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
original-paper
|
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, Christopher Ré
|
primary paper
full_text
|
ID: arXiv:2212.14052v3
sha256:de923022ffc5…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T04:26:45Z
Updated: 2026-10-11T04:26:45Z
Source File:
data/curated/H3Layer.yaml