coCondenser Unsupervised Corpus-Aware Language Model Pre-Training

A two-stage unsupervised language model pre-training algorithm for dense passage retrieval that warm-starts from a universal Condenser model (whose head takes the late-backbone [CLS] representation h_cls^late and short-circuited early-backbone token representations h^early for masked language modeling) and continues pre-training on a target search corpus by sampling two random spans (s_{i1}, s_{i2}) from each of n documents in a batch and minimizing the average sum of Condenser masked language modeling loss and a corpus-aware span contrastive loss L_{ij}^{co} = -log [exp(<h_{i1}, h_{i2}>) / sum_{k=1}^n sum_{l=1}^2 I_{ij != kl} exp(<h_{ij}, h_{kl}>)], using a Gradient Cache v_{ij} = dL^{co}/dh_{ij} to decouple large-batch contrastive gradient computation from sub-batch encoder backpropagation before discarding the Condenser head for small-batch dense retriever fine-tuning.