FastBERT: Self-Distilling BERT with Adaptive Inference Time

A sample-wise adaptive Transformer architecture and three-stage training paradigm that appends a teacher classifier to the final layer L-1 and lightweight student branch classifiers (each comprising a 768->128 projection, 128-dim self-attention, 128->128 dense layer, and N-class softmax) to intermediate layers 0..L-2, fine-tunes the backbone and teacher classifier on task labels with branches disabled, freezes the backbone and teacher to self-distill all L-1 student classifiers by minimizing sum_{i=0}^{L-2} D_KL(p_{s_i}, p_t), and terminates inference for any input sample at layer i when its normalized prediction entropy Uncertainty = (sum_{c=1}^N p_{s_i}(c) log p_{s_i}(c)) / log(1/N) falls below a tunable threshold Speed in [0, 1].