SimLM Representation-Bottleneck Replaced Language Model Pre-Training

A self-supervised pre-training algorithm and bottleneck architecture for dense passage retrieval that corrupts an input passage x twice using a frozen ELECTRA-style generator g with nested replacement probabilities p_enc = 30% and p_dec = 50% (x_enc = Sample(g, Mask(x, p_enc)), x_dec = Sample(g, Mask(x, p_dec)) such that every token replaced in x_enc is also replaced in x_dec), encodes x_enc with a deep Transformer encoder into a single last-layer [CLS] representation bottleneck vector h_cls without skip connections from earlier layers, feeds h_cls and x_dec into a shallow 2-layer bidirectional Transformer decoder, and optimizes the sum of all-position token-level cross-entropy replaced language modeling losses L_pt = L_enc + L_dec where L_enc = -(1/|x|) sum_{i=1}^{|x|} log p(x[i] | x_enc) and L_dec = -(1/|x|) sum_{i=1}^{|x|} log p(x[i] | h_cls, x_dec) before discarding the decoder for multi-stage bi-encoder fine-tuning and KL cross-encoder distillation.