SPARTA Sparse Transformer Matching for Retrieval

A sparse neural retrieval architecture for open-domain and sentence-level question answering that encodes each query q = [t_1, ..., t_|q|] via non-contextualized word embeddings E(q) = [e_1, ..., e_|q|] and each candidate answer with surrounding context (a, c) = [c_1, ..., a_1, ..., a_|a|, c_{a+1}, ..., c_|c|] (distinguished by BERT segment embeddings 1 and 0) via a bidirectional Transformer encoder H(a, c) = [s_1, ..., s_|c|], scores each query token via max-pooled token-level dot-product interaction y_i = max_{j in [1, |c|]}(e_i^T s_j), enforces sparsity and scale regularization through a biased ReLU and logarithm f(q, (a, c)) = sum_{i=1}^{|q|} log(ReLU(y_i + b) + 1), trains end-to-end with a cross-entropy learning-to-rank objective over random and nearby-sentence hard negatives, and precomputes LOOKUP(t, (a, c)) = log(ReLU(max_j(e_t^T s_j) + b) + 1) offline for every vocabulary term t in V to store human-interpretable sparse answer vectors in a standard Inverted Index for O(1) lookup and summation at query time.