TCT-ColBERT Tightly-Coupled Teachers Knowledge Distillation for Dense Retrieval

A knowledge-distillation algorithm for single-stage dense passage retrieval that initializes a Siamese BERT bi-encoder student from a fine-tuned late-interaction ColBERT teacher, removes ColBERT's Conv1D token-projection head to represent each query and passage as a single average-pooled token embedding Pool(E_q), Pool(E_d) in R^h scored via dot product phi_PoolDot(q, d) = <Pool(E_q), Pool(E_d)>, and interleaves online inference of the frozen ColBERT teacher phi_MaxSim(q, d) = sum_{i in |E_q|} max_{j in |E_d|} <eta_q(E_{q_i}), eta_d(E_{d_j})> across all passages D_B = {d^+_{q_j}}_{j in B} U {d^-_{q_j}}_{j in B} in each training mini-batch B to minimize L = -sum_{i=1}^{|B|} [gamma * 1_{d_i in T^+_{q_i}} log P(d_i | q_i) - (1 - gamma) sum_{d' in D_B} KL(P_hat(d' | q_i) || P(d' | q_i))] where P(d | q) = softmax(phi_PoolDot(q, d)) and P_hat(d | q) = softmax(phi_MaxSim(q, d) / tau), optionally combined at inference with top-k sparse retrieval scores via minimum-score-substituted linear interpolation over D_sp U D_ds.