SEED Pre-Training a Strong Text Encoder Using a Weak Decoder

A representation-bottleneck encoder-decoder pre-training architecture for dense text retrieval (SEED-Encoder) that encodes a masked input passage with a deep bidirectional Transformer encoder (trained with masked language modeling) into a single dense [CLS] representation vector and feeds only that [CLS] vector—blocking all non-[CLS] token representations from the encoder—to a structurally weakened autoregressive Transformer decoder (few layers and a restricted local self-attention span k where decoding step i can attend only to the encoder [CLS] vector and at most the preceding k local tokens i-k+1..i) to reconstruct the original uncorrupted document, thereby forcing global semantic information into the encoder [CLS] embedding before discarding the weak decoder for dense retrieval fine-tuning.