Doc2query Neural Document Expansion by Query Prediction
Preprocessing Methods AIMech:Doc2QueryDocumentExpansionByQueryPrediction DRAFT
Aliases: Doc2query, Document Expansion by Query Prediction, Neural Document Expansion by Query Prediction
A neural document-expansion preprocessing method for first-stage information retrieval that trains a sequence-to-sequence Transformer model on (query, relevant document) pairs to generate questions a given document could potentially answer, samples N = 10 predicted queries per document via top-k random sampling prior to indexing, and appends the predicted queries directly to the original document text without special markup before building a standard bag-of-words inverted index (such as BM25), simultaneously performing implicit term re-weighting via copied document tokens (increasing the term frequency of salient words) and bridging lexical vocabulary mismatch via newly generated synonym and related-concept tokens while shifting neural inference cost from retrieval time to offline indexing.
Identity Notes & Concept Scope
Distinct from query-side generation expansion methods such as GAR (AIMech:GARGenerationAugmentedRetrievalForOpenDomainQA) and HyDE (AIMech:HyDEHypotheticalDocumentEmbeddings), which generate expansion contexts from the input query at retrieval time rather than expanding corpus documents offline prior to inverted-index construction; and distinct from DeepCT (AIMech:DeepCTContextAwareTermImportanceEstimationForFirstStageRetrieval), which regresses integer term frequencies for existing document tokens without generating new vocabulary terms not present in the document.
Claims & Grounded Evidence
4 Claims
Mechanism
#doc2query-seq2seq-topk-random-sampling-document-expansion
Doc2query trains a 6-layer encoder-decoder Transformer (512 hidden units, 8 attention heads, 2048 feed-forward units, BPE-tokenized with documents truncated to 400 tokens and queries to 100 tokens, Section 3 & Appendix A) on (query, relevant document) pairs, predicts 10 queries per document using top-k random sampling—which outperforms beam search across all hypothesis counts and peaks at 10 queries before excess queries dilute the original text (Appendix B, Figure 2)—and appends them to the document without special markup prior to BM25 indexing.
Mechanism
#doc2query-complementary-copied-term-reweighting-and-new-term-expansion
Excluding stopwords (51% of predicted query tokens), 69% of Doc2query's predicted words exist in the source document (copied words that perform implicit term re-weighting by boosting key term frequencies) while 31% are novel words absent from the source document (new words that mitigate vocabulary mismatch via synonyms and related terms): on MS MARCO Dev, expanding with new words alone improves BM25 MRR@10 from 18.4 to 18.8, copied words alone reaches 19.7, and combining both reaches 21.5 MRR@10 and raises Recall@1000 from 85.3 to 89.3 (Section 5, Tables 1-2).
Application
#doc2query-msmarco-and-treccar-retrieval-and-reranking-application
Doc2query improves standalone BM25 retrieval effectiveness by ~15% on both TREC-CAR (15.3 to 18.3 MAP) and MS MARCO (18.6 to 21.8 Test MRR@10 at 90ms/query vs. 650ms/query for Single Duet v2), outperforms RM3 query expansion (which degrades MS MARCO Dev MRR@10 to 16.7), and provides orthogonal gains when pipelined with BERT cross-encoder reranking (36.5 MAP on TREC-CAR and 37.5 Dev / 36.8 Test MRR@10 on MS MARCO, Table 1).
Limitation
#doc2query-corpus-reindexing-and-overexpansion-noise-limitation
Because document expansion modifies corpus representations prior to indexing, any change to the expansion model requires running neural generation across all corpus documents and re-indexing the entire collection (Section 2), and appending too many predicted queries (e.g., 20 queries in Figure 2) introduces noise that reduces the relative contribution of the original document text and degrades MRR@10 (Appendix B).
⚠️ Curated Limitations & Negative Results
- Modifying the document expansion model requires re-generating queries for every document in the corpus and re-building the inverted index from scratch.
- Appending more than ~10 predicted queries per document introduces noisy terms that dilute the original document text and reduce retrieval effectiveness.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | Doc2query trains a 6-layer encoder-decoder sequence-to-sequence Transformer neural network to generate synthetic expansion queries for corpus documents prior to indexing. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r773-primary-pdf
|
Rodrigo Nogueira, Wei Yang, Jimmy Lin, Kyunghyun Cho
|
primary paper
full_text
|
ID: https://arxiv.org/bibtex/1904.08375
sha256:ae7ba200f53a…
|
r773-arxiv-abs
|
Rodrigo Nogueira, Wei Yang, Jimmy Lin, Kyunghyun Cho
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1904.08375
sha256:6d7b7cdaa93f…
|
r773-metadata-cr
|
Rodrigo Nogueira, Wei Yang, Jimmy Lin, Kyunghyun Cho
|
primary paper
abstract_only
|
ID: https://arxiv.org/bibtex/1904.08375
sha256:d547fbca2f6c…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T14:16:00Z
Updated: 2026-10-19T14:16:00Z
Source File:
data/curated/Doc2QueryDocumentExpansionByQueryPrediction.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories