PROP Pre-Training with Representative Words Prediction for Ad-Hoc Retrieval

A self-supervised pre-training algorithm tailored for ad-hoc document retrieval that estimates a Dirichlet-prior-smoothed multinomial unigram language model P(w | D) = (c(w, D) + mu P(w | C)) / (|D| + mu) for each document D in an unlabeled corpus C, samples a word-set length l ~ Poisson(lambda) and draws a pair of equal-length word sets (S_1, S_2) independently from P(w | D), labels the word set with higher query likelihood P(S | theta_D) = prod_{w in S} P(w | D) as more representative of D (say S_1), encodes each concatenated sequence [CLS] + S_i + [SEP] + D + [SEP] with an L-layer Transformer and an MLP head over H^{[CLS]} to predict representativeness score s(S_i | D), and jointly minimizes the Representative Words Prediction (ROP) pairwise hinge loss L_ROP = max(0, 1 - s(S_1 | D) + s(S_2 | D)) and Masked Language Modeling loss L_MLM.