Home / Curated / Architectures / SMITH: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching

SMITH: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching

Architectures AIMech:SMITHSiameseMultiDepthTransformerBasedHierarchicalEncoder DRAFT

Aliases: SMITH Hierarchical Document Matching Encoder, Siamese Multi-depth Transformer-based Hierarchical Encoder, SMITH Dual-Encoder for Long-Form Document Matching

A two-tower Siamese ('Dual-Encoder') hierarchical Transformer architecture for long-to-long document semantic matching (Yang, Zhang, Li, Bendersky, & Najork, CIKM 2020) that packs natural sentences into L_d sentence blocks {S_1, ..., S_{L_d}} of max length L_s via greedy sentence filling, encodes tokens within each block using an L_1-layer sentence-level Transformer to extract L2-normalized block [CLS] embeddings h_i in R^H, contextualizes {h_1, ..., h_{L_d}} with an L_2-layer document-level Transformer to produce L2-normalized document embeddings E(d_s), E(d_c) scored by cosine similarity cos(E(d_s), E(d_c)), and pre-trains jointly with masked word prediction and dynamic in-batch masked sentence block contrastive prediction L_pretrain = L_sp + L_wp.

Identity Notes & Concept Scope Distinct from cross-encoder passage aggregation rankers such as PARADE (AIMech:PARADEPassageRepresentationAggregationForDocumentReranking, which jointly encodes query-passage pairs) and from RNN-based hierarchical document matchers such as HAN and SMASH RNN. SMITH is a representation-focused Siamese dual encoder supporting offline vector indexing up to 2,048 tokens, reducing self-attention memory complexity from O(b * A * L * N^2) to O(b * A * L * (L_s * N + (N / L_s)^2)) via greedy sentence filling and two-level Transformers.

Claims & Grounded Evidence

3 Claims

Mechanism #smith-greedy-sentence-filling-and-two-level-transformer-mechanism

SMITH splits a document of length N into L_d = N / L_s blocks of max length L_s by greedily packing complete natural sentences into each block until adding the next sentence would exceed L_s, encodes each block with an L_1-layer sentence-level Transformer, and feeds the L2-normalized block representations plus block position embeddings into an L_2-layer document-level Transformer, reducing attention memory complexity from O(b * A * L * N^2) to O(b * A * L * (L_s * N + (N / L_s)^2)) (Eq. 3).

supports Source: Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching · Locator: Section 4.1.1-4.1.3, Equations (1)-(3), Figures 2-3, pp. 4-6
supports Source: Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching · Locator: greedy_sentence_filling, count_padding_tokens, and smith_attention_memory_ops verification

Mechanism #smith-dynamic-masked-sentence-block-pretraining-mechanism

During unsupervised pre-training, SMITH dynamically masks m randomly sampled sentence block vectors h_k per document with a learned mask vector h_hat, predicts contextual block vectors S_hat_j from the document-level Transformer, and minimizes the in-batch softmax cross-entropy L_sp = -(1/B) sum_{i=1}^B log(exp(S_hat_i h_i^T) / sum_{r=1}^B exp(S_hat_i h_r^T)) over all B = b * m masked blocks jointly with masked word prediction L_pretrain = L_sp + L_wp (Eqs. 4-7).

supports Source: Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching · Locator: Section 4.2.1-4.2.2, Equations (4)-(7), p. 6
supports Source: Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching · Locator: masked_sentence_block_loss verification

Limitation #smith-siamese-no-early-cross-document-token-interaction-limitation

Because SMITH is a representation-focused Siamese dual encoder that encodes source and candidate documents independently into single vectors for offline ANN indexing, it cannot perform early token-by-token cross-attention between the two documents and requires tuning sentence-level Transformer depth L_1 against intermediate activation memory costs.

supports Source: Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching · Locator: Section 2, p. 3; Section 5.2.1, p. 8; Section 5.5, Table 6, p. 9

⚠️ Curated Limitations & Negative Results

As a Siamese dual-encoder architecture that compresses each document into a single dense vector before cosine similarity, SMITH cannot capture fine-grained cross-document token interactions prior to final pooling.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	broad_match	SMITH is a Siamese hierarchical Transformer neural network architecture for long-form document representation learning and semantic matching.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r790-primary-pdf`	Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, Marc Najork	primary paper full_text	ID: https://arxiv.org/bibtex/2004.12297 sha256:b9608127f1ee…
`r790-arxiv-abs`	Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching (arXiv abstract) Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, Marc Najork	primary paper abstract_only	ID: https://arxiv.org/abs/2004.12297 sha256:61741c16b448…
`r790-metadata-cr`	Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching (Crossref metadata) Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, Marc Najork	primary paper abstract_only	ID: https://arxiv.org/bibtex/2004.12297 sha256:d8711a695cf8…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T14:33:00Z

Updated: 2026-10-19T14:33:00Z

Source File: data/curated/SMITHSiameseMultiDepthTransformerBasedHierarchicalEncoder.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories