Home / Curated / Algorithms / LM-BFF Better Few-Shot Fine-Tuning of Language Models

LM-BFF Better Few-Shot Fine-Tuning of Language Models

Algorithms AIMech:LMBFFBetterFewShotFineTuning DRAFT

Aliases: LM-BFF, Better Few-Shot Fine-Tuning of Language Models, Language Models Best Friends Forever Few-Shot Fine-Tuning, LM-BFF Automated Prompt and Demonstration Fine-Tuning

A few-shot fine-tuning framework for moderately sized pretrained masked language models that reformulates classification and bounded-interval regression (via two-pole mixture interpolation y = v_l p(y_l|x_in) + v_u p(y_u|x_in)) as cloze auto-completion p(y|x_in) = p([MASK]=M(y)|T(x_in)), automates prompt construction by pruning and re-ranking conditional label-word assignments M(Y) alongside beam-search span-infilling template generation with a pretrained T5 text-to-text model maximizing sum_{(x_in,y) in D_train} log P_T5(T|T_g(x_in,y)), and dynamically concatenates one demonstration example per class sampled from the top-r% most semantically similar training instances cos(e(x_in), e(x)) under a sentence encoder.

Identity Notes & Concept Scope Distinct from Pattern-Exploiting Training (PET, Schick & Schütze, EACL 2021, AIMech:PatternExploitingTraining), which uses manually crafted patterns and unlabeled data distillation, from AutoPrompt (Shin et al., EMNLP 2020, AIMech:AutoPromptGradientGuidedDiscretePromptSearch), which performs gradient-guided discrete trigger token search on frozen MLMs requiring large training sets, and from frozen GPT-3 In-Context Learning (AIMech:InContextLearning), which concatenates up to 32 random examples without parameter updates or class-balanced similarity filtering.

Claims & Grounded Evidence

4 Claims

Mechanism #r719-claim-two-pole-regression-and-mlm

LM-BFF casts classification as masked token prediction p(y|x_in) = exp(w_{M(y)} . h_{[MASK]}) / sum_{y'} exp(w_{M(y')} . h_{[MASK]}) without adding new classification head parameters, and casts bounded-interval regression over [v_l, v_u] as a two-pole mixture interpolation y = v_l p(y_l|x_in) + v_u p(y_u|x_in) trained by minimizing the KL divergence between inferred p(y_u|x_in) and observed mixture weight (y - v_l)/(v_u - v_l).

supports Source: Making Pre-trained Language Models Better Few-shot Learners · Locator: Section 4.1-4.2, Equations 1-2, pp. 4-5

Mechanism #r719-claim-t5-template-and-label-search

To automate prompt design from only K=16 examples per class, LM-BFF prunes candidate label words per class via zero-shot conditional likelihood Top-k_{v in V} sum_{x_in in D_train^c} log P_L([MASK]=v|T(x_in)) followed by fine-tuning re-ranking on D_dev, and decodes diverse natural templates via wide-beam search over a pretrained T5 span-infilling model maximizing sum_{j=1}^{|T|} sum_{(x_in,y) in D_train} log P_T5(t_j | t_{1..j-1}, T_g(x_in,y)).

supports Source: Making Pre-trained Language Models Better Few-shot Learners · Locator: Section 5.1-5.2, Equations 3-4, Figure 2, pp. 5-6

Application #r719-claim-selective-demonstration-sampling

By concatenating one filled template demonstration T_tilde(x_in^(c), y^(c)) per class sampled from the top r=50% most cosine-similar training examples cos(e(x_in), e(x)) in SBERT embedding space, LM-BFF with RoBERTa-large (K=16 per class) outperforms standard few-shot fine-tuning by 11% on average and up to 30% absolute on SNLI (reaching 79.7% vs. 48.4%) across 15 classification and regression benchmarks.

supports Source: Making Pre-trained Language Models Better Few-shot Learners · Locator: Section 6.1-6.2 & Section 7.1-7.3, Tables 3 and 7, pp. 6-9

Limitation #r719-claim-context-length-and-cola-limitation

Because LM-BFF concatenates one demonstration per class into the input context, it is constrained by the model's maximum sequence length (e.g., 512 tokens in RoBERTa) when input sequences are long or the number of classes |Y| is large, and prompt-based fine-tuning underperforms standard fine-tuning on linguistic acceptability (CoLA, 21.8 vs. 33.9 Matthews correlation) where ungrammatical inputs lie out of the masked language model's pretraining distribution.

supports Source: Making Pre-trained Language Models Better Few-shot Learners · Locator: Section 7.1 & Section 8, Table 3, pp. 7-9

⚠️ Curated Limitations & Negative Results

Concatenating one demonstration per class into the input context is bounded by the backbone's maximum context window (e.g., 512 tokens), restricting applicability to tasks with short inputs and a modest number of output classes.
Underperforms standard fine-tuning on out-of-distribution grammatical acceptability tasks such as CoLA (21.8 vs. 33.9 at K=16) and still requires either a manual seed template (for label word search) or manual seed label words (for T5 template generation).

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	LM-BFF adapts pretrained Transformer masked language models to few-shot classification and regression tasks via automated template generation and selective demonstration fine-tuning.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r719-primary-pdf`	Making Pre-trained Language Models Better Few-shot Learners Tianyu Gao, Adam Fisch, Danqi Chen	primary paper full_text	ID: https://dblp.org/search/publ/api?q=Making+Pre-trained+Language+Models+Better+Few-shot+Learners+Gao&format=xml sha256:e99b70d86c61…
`r719-arxiv-abs`	Making Pre-trained Language Models Better Few-shot Learners (arXiv abstract) Tianyu Gao, Adam Fisch, Danqi Chen	primary paper abstract_only	ID: https://arxiv.org/abs/2012.15723 sha256:00c5c1ef2c54…
`r719-metadata-cr`	Making Pre-trained Language Models Better Few-shot Learners (Crossref metadata) Tianyu Gao, Adam Fisch, Danqi Chen	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=Making+Pre-trained+Language+Models+Better+Few-shot+Learners+Gao&format=xml sha256:1255530331e7…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:22:00Z

Updated: 2026-10-19T13:22:00Z

Source File: data/curated/LMBFFBetterFewShotFineTuning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories