Condenser (Pre-Training Architecture for Dense Retrieval)

A transformer pre-training architecture for dense bi-encoder representation learning that partitions the encoder backbone into L^e early layers and L^l late layers—computing [h_cls^early; h^early] = Encoder_early([h_cls^0; h^0]) and [h_cls^late; h^late] = Encoder_late([h_cls^early; h^early])—and introduces a short-circuit from the early token states h^early combined with the late [CLS] vector h_cls^late into a temporary L^h-layer Condenser Head [h_cls^cd; h^cd] = Condenser_head([h_cls^late; h^early]) trained with joint Masked Language Modeling loss L = L_mlm(W h_i^cd, x_i) + L_mlm^c(W h_i^late, x_i) before discarding the head during fine-tuning.