Contrastive predictive coding

Compute z_t=g_enc(x_t) and c_t=g_ar(z_<=t). For each selected future offset k use the positive score f_k(x,c_t)=exp(z(x)^T W_k c_t). Form a candidate set containing one true future observation from the context-conditional distribution and N−1 negatives, ideally drawn independently from the observation marginal. Minimize the expected negative log of the positive score divided by the sum of all N scores, including the positive, over selected prediction horizons. Train the encoder, context module and prediction parameters jointly. With the ideal sampling scheme and a sufficiently expressive optimally fitted score, the optimal score is proportional to the conditional-to-marginal density ratio, with a context-dependent factor cancelling in the categorical probabilities. The expected loss yields the InfoNCE lower bound I(X_future;C)>=log(N)−L_NCE under its sampling assumptions. This is an expectation-level statement, not a guarantee for one minibatch. Learned local representations or context summaries can be used downstream.