DensePhrases Dense Phrase Retrieval at Scale

A reader-free, purely dense phrase-retrieval architecture and training framework for open-domain question answering and knowledge-intensive NLP that encodes each passage p = w_1, ..., w_m with a SpanBERT phrase encoder M_p into token vectors h_1, ..., h_m in R^d to represent any contiguous phrase s_k up to L words by concatenating its start and end token vectors E_s(s_k, p) = [h_{start(k)}, h_{end(k)}] in R^{2d} while storing only filtered word vectors H in R^{|W(D)| x d}, encodes each question q via two separate encoders M_{q,start} and M_{q,end} into E_q(q) = [q^start, q^end] in R^{2d} for constrained maximum inner product search s_hat = argmax_{1 <= i <= j < i+L} (H q^start)_i + (H q^end)_j, trains phrase and question representations jointly with single-passage span log-likelihood, cross-attention KL distillation, in-batch negatives, and FIFO-cached pre-batch negatives from the preceding C mini-batches (L = lambda_1 L_single + lambda_2 L_distill + lambda_3 L_neg), and performs query-side fine-tuning by updating only E_q to maximize the marginal log-likelihood of gold answer strings among top-k retrieved phrases from the fixed phrase index H.