CEDR Contextualized Embeddings for Document Ranking

CEDR expands the 2D query-document similarity matrix S in R^{|Q| x |D|} into a 3D multi-layer similarity tensor S_{Q,D}[l, q, d] = cos(context_{Q,D}(q, l), context_{Q,D}(d, l)) in R^{L x |Q| x |D|} across all L layers of a contextualized language model (Section 2.2, Equation 1): unlike static GloVe embeddings where identical tokens always yield cosine similarity 1.0, S_{Q,D}[l, q, d] assigns context-dependent similarity (< 1.0) that separates matching word senses (e.g., 'curb' meaning restrain population growth vs. street construction in Figure 1) and feeds each layer l as an input channel to PACRR convolutions or KNRM/DRMM counting kernels.

To combine deep next-sentence semantic classification with token-level relevance matching, CEDR encodes the query (Segment A) and document segments (Segment B, split evenly within BERT's 512-token limit and averaged across splits) simultaneously through BERT and fuses the [CLS] classification vector with the neural ranker—by concatenating [CLS] to the multi-layer kernel/n-gram signals before dense combination in CEDR-KNRM and CEDR-PACRR, or by including [CLS] in the per-query-term dense combination of bin scores in CEDR-DRMM (Sections 2.3 & 3.1).

On TREC Robust 2004 and WebTrack 2012-14, CEDR-PACRR, CEDR-KNRM, and CEDR-DRMM all significantly outperform BM25, ConvKNRM, GloVe-based rankers, Vanilla BERT, and their non-CEDR contextualized counterparts (Table 1, reaching 0.4667 P@20 and 0.5381 nDCG@20 with CEDR-KNRM on Robust04 vs. 0.4042 / 0.4541 for Vanilla BERT), while truncating BERT execution at layer 5 retains nearly all of CEDR-PACRR's nDCG@20 / ERR@20 gains at over 2x faster inference speed (Section 3.2, Figure 2b).

Because CEDR jointly encodes the query and document through BERT at query time and computes |Q| x |D| cosine similarity matrices across all L layers, its inference throughput is orders of magnitude slower than static GloVe-based interaction rankers (Figure 2a), and documents exceeding BERT's 512-token limit up to the 800-token truncation cap must be split into multiple segments and processed separately (Sections 3.1-3.2).