DRAGON (Diverse Augmentation for Generalizable Dense Retrieval)

Given N heterogeneous teacher retrievers ordered by increasing zero-shot generalization capability (uniCOIL -> Contriever -> GTR-XXL -> ColBERTv2 -> SPLADE++), DRAGON trains an asymmetric BERT-base bi-encoder across N = 5 progressive iterations; at iteration T <= N, for each augmented query q (sampled from 28M cropped MS MARCO sentences and 28M DocT5Query synthetic queries), it uniformly samples a teacher n ~ U(1, T), selects a positive d^+ from the top k = 10 passages and a hard negative d^- from ranks 46..50 of P_q^n (assigning consensus positives retrieved by m <= T teachers sampling probability m / (T*k) without score normalization), and minimizes InfoNCE loss.

Without increasing inference model size beyond a 110M-parameter BERT-base bi-encoder or requiring target-domain corpora during training, DRAGON and DRAGON+ (initialized from RetroMAE) achieve 39.3 and 39.0 MRR@10 on supervised MS MARCO Dev, 74.1 and 74.4 nDCG@10 on TREC DL 2019, and 49.4 and 50.2 average nDCG@10 on zero-shot BEIR-13 (matching SPLADE++'s 50.3 and surpassing ColBERTv2's 49.9 and GTR-XXL's 49.3), while reversing the progressive teacher trajectory degrades BEIR-13 by 1.6 points.

DRAGON requires offline batch retrieval over 28M augmented queries across five distinct sparse, dense, and multi-vector teacher indices followed by five 20-epoch training iterations (around 650K total optimization steps on 32 A100 GPUs), is sensitive to the easy-to-hard ordering of teacher retrievers, and still trails late-interaction/sparse teachers (SPLADE++ and ColBERTv2) on specialized science and technology forum queries in LoTTE.