Home / Curated / Algorithms / T-Few Fine-Tuning

T-Few Fine-Tuning

Algorithms AIMech:TFewFineTuning DRAFT

Aliases: T-Few

A few-shot classification fine-tuning recipe that adapts a pretrained T0 model through IA3 activation vectors, multitask pretraining of those vectors and combined language-modeling, unlikelihood and length-normalized candidate objectives, with documented task-specific exceptions.

Identity Notes & Concept Scope Compared all 442 pinned AIO and 632 native IDs, names, aliases and scoped synonyms; no normalized identity match. Ignored- and hidden-inclusive fixed-string searches of the recovery, older discovery and original Documents checkouts returned 153, 0, 0 lines. All distinct match contents were inspected. Semantic matching covered 40 AIO and 92 native definitions, with every matched definition read. 7 full annotated AIO node/edge/generated contexts and 3 full native records were inspected. Retain the composite T-Few recipe, separately from IA3 activation scaling, Adafactor optimization, generic few-shot fine-tuning, gradient-free in-context learning and a trained T0 checkpoint.

Claims & Grounded Evidence

5 Claims

Mechanism #adaptation-stages

The recipe begins with a pretrained T0 backbone, adds IA3 key/value/feedforward scaling vectors and pretrains the added vectors on the T0 multitask mixture for 100,000 steps with batch size 16. Downstream task adaptation trains the added parameters for 1000 steps with batch size 8 using Adafactor, a learning-rate setting of 3e-3, linear decay and 60 warmup steps. The base weights are frozen during this parameter-efficient adaptation; the pretrained backbone itself required earlier training.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Sections 3.3–3.5, main PDF pp. 6–7

Mechanism #token-level-unlikelihood

The unlikelihood term averages negative log(1-p) over tokens of all incorrect candidate strings, with each conditional token probability evaluated under that incorrect string’s preceding tokens and the input. It is not negative log of one minus the probability of an entire incorrect sequence. A token shared by a correct and an incorrect candidate can receive opposing language-model and unlikelihood contributions.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Section 3.2, main PDF p. 5, Equation 1; constructed token/sequence example

Mechanism #length-normalized-candidate-objective

For each candidate string, beta is its mean conditional token log probability. A softmax over beta scores produces a candidate classification loss for the correct string. Exponentiating beta gives a geometric mean of token probabilities, not a normalized whole-sequence probability. The ordinary correct-string language-model loss, token unlikelihood loss and length-normalized candidate loss are summed with unit coefficients in the stated general recipe.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Section 3.2, main PDF p. 5, Equation 2 and following paragraph; constructed ranking and derivative example

Limitation #banking77-exception

The supplemental RAFT protocol turns off the unlikelihood term on Banking77 because its 77 classes cause memory issues. It also includes all labels in that task’s input and replaces periods in labels with commas. These documented changes qualify the main paper’s broad description of a recipe used without task-specific modifications; an unchanged three-loss recipe for every reported task is not supported.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — Supplementary Material · Locator: Appendix G, supplemental PDF p. 2/printed 18

Scope #evaluation-and-cost-boundaries

The recipe was designed using T0-3B and evaluated with a larger T0 backbone on configured few-shot classification and RAFT tasks. Comparisons use sampled subsets and prompts; imported GPT-3 results do not use the same known training examples or the same backbone. Reported aggregate accuracy, adaptation FLOPs and historical hardware costs do not imply superiority on every task, universal latency savings or current service prices. Generative tasks are identified as future work.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Sections 3.1 and 4–5, main PDF pp. 4–5 and 8–10
supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — Supplementary Material · Locator: Appendices E–G and Tables 8–11, supplemental PDF pp. 2 and 7–8/printed 18 and 23–24

⚠️ Curated Limitations & Negative Results

T-Few is a composite training and prediction recipe. IA3 parameterization, Adafactor optimization, the T0 checkpoint and gradient-free in-context learning are distinct identities.
The documented Banking77 exception must be retained alongside the default objective. The source’s broad unchanged-recipe wording does not override its detailed protocol.
Candidate length normalization alters rankings and training gradients; it does not remove all prompt, tokenization, class-count or label-string effects.
Complete sixteen-page main paper and eight-page supplement visually inspected. All empirical results are source reports; no author code, model fitting, paid provider, hardware timing or benchmark replication was executed.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:KnowledgeTransfer` Knowledge Transfer	related_match	The recipe reuses a pretrained multitask representation and adaptation initialization for a new task; the generic transfer strategy is related context rather than an equivalent recipe.
`AIO:DeepTransferLearning` Deep Transfer Learning	related_match	T-Few adapts a deep language-model representation. The pinned target denotes a network category, so it is not asserted to be an algorithm superclass.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`liu2022ia3`	Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel	primary paper full_text	ID: doi:10.52202/068431-0142 sha256:9c107aa36f12…
`liu2022ia3supp`	Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — Supplementary Material Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel	primary paper full_text	ID: doi:10.52202/068431-0142 sha256:0da202641e40…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-13T16:41:55Z

Updated: 2026-10-13T16:41:55Z

Source File: data/curated/TFewFineTuning.yaml