HIBERT (Hierarchical Bidirectional Transformers for Document Summarization)
Architectures AIMech:HIBERTHierarchicalBidirectionalTransformersForDocumentSummarization DRAFT
Aliases: HIBERT, Hierarchical Bidirectional Transformers for Document Summarization, HIBERT Document-Level Pre-trained Hierarchical Transformer
A hierarchical bidirectional Transformer document encoding and pre-training framework (Zhang, Wei, & Zhou, ACL 2019) that stacks a word-level sentence Transformer encoder producing sentence vectors from per-sentence end-of-sentence positions and a document-level Transformer encoder computing context-sensitive sentence representations across an entire document, pre-trained on unlabeled documents via a sentence-level masking and autoregressively decoded masked-sentence prediction objective before fine-tuning for extractive document summarization.
Identity Notes & Concept Scope
Distinct from flat BERT/RoBERTa encoders (which concatenate tokens into a single sequence and pre-train with token-level MLM and next-sentence prediction) and from SMITH (AIMech:SMITHSiameseMultiDepthTransformerBasedHierarchicalEncoder, which replaces autoregressive masked-sentence decoding with masked sentence-block projection classification in a Siamese dual-encoder retrieval architecture). HIBERT pre-trains a two-level hierarchical Transformer encoder by masking 15% of sentences (80% all-[MASK] tokens, 10% unchanged, 10% random sentence replacement) and predicting each masked sentence word-by-word via an auxiliary Transformer decoder that injects the context-sensitive sentence representation d_tilde_k into the two-layer feed-forward network after multi-head self-attention.
Claims & Grounded Evidence
4 Claims
Mechanism
#hibert-hierarchical-sentence-and-document-encoders
HIBERT encodes a document D = (S_1, S_2, ..., S_|D|) with a sentence Transformer encoder E_S that maps each sentence S_k = (w_1^k, ..., w_|S_k|^k) (terminated by an end-of-sentence token) to hidden vectors (h_1^k, ..., h_|S_k|^k) and selects the final hidden state s_k = h_|S_k|^k as the sentence representation, followed by a document Transformer encoder E_D that adds sentence positional embeddings p_k to form d_k = s_k + p_k and applies multi-layer bidirectional self-attention across (d_1, ..., d_|D|) to obtain context-sensitive sentence representations (d_tilde_1, ..., d_tilde_|D|).
Mechanism
#hibert-sentence-masking-and-context-injected-decoder
To pre-train HIBERT on unlabeled documents, 15% of sentences in D are selected at random (80% masked by replacing every word with [MASK], 10% kept unchanged, and 10% replaced with a random sentence from another document), and an auxiliary Transformer decoder predicts each selected sentence S_k word-by-word by injecting its context-sensitive sentence vector d_tilde_k into the intermediate representation h_tilde_j^k after masked multi-head attention via x_tilde_j^k = h_tilde_j^k + d_tilde_k and g_tilde_j^k = W_2^ff max(0, W_1^ff x_tilde_j^k + b_1) + b_2 (Equations 4-6).
Application
#hibert-extractive-summarization-empirical-gains
When fine-tuned as a hierarchical extractive summarizer predicting binary sentence labels Y = (y_1, ..., y_|D|), HIBERT_M (12+12 layers) achieves 42.37 ROUGE-1, 19.95 ROUGE-2, and 38.83 ROUGE-L on CNN/DailyMail (+1.25 ROUGE-1 and +1.25 ROUGE-2 over the non-pre-trained HIER-BERT baseline) and 49.47 ROUGE-1, 26.17 ROUGE-2, and 46.69 ROUGE-L on New York Times50.
Limitation
#hibert-autoregressive-sentence-decoding-pretraining-cost
Pre-training HIBERT requires an auxiliary autoregressive Transformer decoder that predicts every masked sentence token-by-token, incurring higher pre-training compute and memory overhead than non-autoregressive masked sentence-block projection objectives and requiring sentences to be truncated to 50 words and documents to 30 sentences during training.
⚠️ Curated Limitations & Negative Results
- Pre-training requires an auxiliary autoregressive Transformer decoder over masked sentences and truncates documents to 30 sentences of at most 50 words each.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
broad_match | HIBERT is a specialized hierarchical two-level bidirectional Transformer neural network architecture for document encoding and extractive summarization. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r791-primary-pdf
|
Xingxing Zhang, Furu Wei, Ming Zhou
|
primary paper
full_text
|
ID: https://arxiv.org/bibtex/1905.06566
sha256:465bec71c3e7…
|
r791-arxiv-abs
|
Xingxing Zhang, Furu Wei, Ming Zhou
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1905.06566
sha256:85c487ce0e08…
|
r791-metadata-cr
|
Xingxing Zhang, Furu Wei, Ming Zhou
|
primary paper
abstract_only
|
ID: https://arxiv.org/bibtex/1905.06566
sha256:2e808b56f3f4…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T14:34:00Z
Updated: 2026-10-19T14:34:00Z
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories