TKL: Transformer-Kernel Model with Local Self-Attention for Long Document Retrieval

A long-document neural re-ranking architecture (Hofstätter, Zamani, Mitra, Craswell, & Hanbury, SIGIR 2020) that contextualizes document token sequences d_{1:m} (up to thousands of tokens) in fixed-size moving windows of width w with overlap o using a multi-layer highway Transformer—reducing self-attention complexity from O(L^2) to O(L * w) and skipping pure-padding windows—computes cosine similarity RBF kernel activations K_{i,j}^k = exp(-(cos(q_i, d_j) - mu_k)^2 / (2 sigma^2)), sums activations within sliding saturation regions K_i^{r,k} = sum_{j=r}^{r+r_size} K_{i,j}^k, applies a learned query-salience conditioned power saturation function K_hat_i^{r,k} = a_i * (K_i^{r,k})^{1/b_i} - c_i, and aggregates the resulting relevance topography s^r via top-local-max_{t,f}(s^r) W_s over the top-t non-overlapping local maxima and their f immediate neighbors.