Home / Curated / Architectures / QDS-Transformer: Query-Directed Sparse Transformer for Long Document Ranking

QDS-Transformer: Query-Directed Sparse Transformer for Long Document Ranking

Architectures AIMech:QDSTransformerQueryDirectedSparseTransformerForLongDocumentRanking DRAFT

Aliases: Query-Directed Sparse Transformer, QDS-Transformer Long Document Ranker, Query-Directed Sparse Attention Transformer

A long-document ranking Transformer architecture (Jiang, Xiong, Lee, & Wang, Findings of EMNLP 2020) that replaces the dense n x n self-attention adjacency matrix A = 1 in H_hat^l = A * M * V^T with a Query-Directed Sparse adjacency matrix A_QDS = A_local U A_sent U A_query U A_[CLS] over the concatenated query-document sequence (up to n = 2,048 tokens), combining local sliding-window attention (A_local[i, j] = 1 iff |i - j| <= w/2), hierarchical sentence-boundary attention on prepended start-of-sentence tokens (A_sent[i, j] = 1 iff t_j = [SOS]), query-directed global attention on query tokens (A_query[i, j] = 1 iff t_i in q), and global [CLS] attention, reducing per-layer self-attention complexity from O(n^2 * dim) to O(n * dim * (w + |q| + |s|)).

Identity Notes & Concept Scope Distinct from general-purpose sparse Transformers such as Sparse Transformer (local/strided attention only) and Longformer-QA (which places global attention only on two task tokens without sentence-level [SOS] hierarchy or full query-directed paths), and from passage-splitting rankers such as BERT-MaxP or PARADE. QDS-Transformer encodes the entire long document up to 2,048 tokens in a unified cross-encoder with IR-axiomatic sparse attention compiled via TVM CUDA kernels.

Claims & Grounded Evidence

3 Claims

Mechanism #qds-transformer-sparse-adjacency-union-mechanism

QDS-Transformer prepends a special [SOS] token to every sentence in the document and defines sparse self-attention adjacency A_QDS = A_local U A_sent U A_query U A_[CLS] (Eq. 10), where A_local[i, j] = 1 iff |i - j| <= w/2 (Eq. 7, e.g., w = 128), A_sent[i, j] = 1 iff t_j = [SOS] (Eq. 8), and A_query[i, j] = 1 iff t_i in q (Eq. 9), reducing self-attention complexity from O(n^2 * dim) to O(n * dim * (w + |q| + |s|)).

supports Source: Long Document Ranking with Query-Directed Sparse Transformer · Locator: Section 4.1-4.3, Equations (7)-(10), p. 4
supports Source: Long Document Ranking with Query-Directed Sparse Transformer · Locator: build_qds_adjacency vs build_local_only_adjacency sparsity and reachability verification

Scope #qds-transformer-long-document-ranking-and-ablation-scope

On TREC 2019 Deep Learning Track document ranking and few-shot TREC benchmarks (Robust04, ClueWeb09-B, ClueWeb12-B13), QDS-Transformer outperforms RoBERTa-MaxP, TKL, Sparse-Transformer, Longformer-QA, and Transformer-XH while achieving over 2x speedup compared to dense 2,048-token RoBERTa when compiled with TVM custom CUDA kernels.

supports Source: Long Document Ranking with Query-Directed Sparse Transformer · Locator: Section 6.1-6.3, Tables 2-5, Figure 3, pp. 6-7

Limitation #qds-transformer-irregular-sparsity-cuda-kernel-limitation

Realizing actual wall-clock GPU speedups from A_QDS requires custom Tensor Virtual Machine (TVM) CUDA kernel compilation (as naive masking is slower than dense self-attention), and irregular sentence-boundary skipping patterns in A_sent and A_query achieve lower hardware utilization than purely diagonal local windows while token-wise feed-forward layers O(dim^2 * n) remain unpruned.

supports Source: Long Document Ranking with Query-Directed Sparse Transformer · Locator: Section 4.3, pp. 4-5; Section 6.3, Table 5, p. 7

⚠️ Curated Limitations & Negative Results

Requires custom TVM CUDA kernels to avoid naive masking overhead, and O(dim^2 * n) feed-forward layers remain a computational bottleneck even when attention is sparse.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	broad_match	QDS-Transformer is a sparse-attention Transformer neural network architecture tailored for long document ranking.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r789-primary-pdf`	Long Document Ranking with Query-Directed Sparse Transformer Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee, Wei Wang	primary paper full_text	ID: https://arxiv.org/bibtex/2010.12683 sha256:b4c4785bdb9c…
`r789-arxiv-abs`	Long Document Ranking with Query-Directed Sparse Transformer (arXiv abstract) Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee, Wei Wang	primary paper abstract_only	ID: https://arxiv.org/abs/2010.12683 sha256:e94fcd4fb163…
`r789-metadata-cr`	Long Document Ranking with Query-Directed Sparse Transformer (Crossref metadata) Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee, Wei Wang	primary paper abstract_only	ID: https://arxiv.org/bibtex/2010.12683 sha256:a06a3c682232…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T14:32:00Z

Updated: 2026-10-19T14:32:00Z

Source File: data/curated/QDSTransformerQueryDirectedSparseTransformerForLongDocumentRanking.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories