Home / Curated / Algorithms / Lottery Ticket Sparse Fine-Tuning

Lottery Ticket Sparse Fine-Tuning

Algorithms AIMech:LTSFTLotteryTicketSparseFineTuning DRAFT

Aliases: LT-SFT, Composable Lottery-Ticket Sparse Fine-Tuning, Composable Sparse Fine-Tuning for Cross-Lingual Transfer

A modular two-phase sparse fine-tuning algorithm for pretrained language models inspired by the Lottery Ticket Hypothesis that first performs full fine-tuning from pretrained parameters theta^(0) on task or language data to obtain theta^(1) and selects a binary mask mu over the K parameters with greatest absolute deviation |theta_i^(1) - theta_i^(0)|, then resets the model to its original pretrained initialization theta^(0) (without zeroing unselected weights) and re-trains only the masked parameters via masked gradients mu odot nabla_theta L to produce a sparse difference vector phi = theta^(2) - theta^(0) that can be composed additively across languages and tasks as F(.; theta^(0) + phi_T + phi_L) without modifying model architecture or increasing inference cost.

Identity Notes & Concept Scope Introduced by Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulic (ACL 2022, pp. 1778-1796, arXiv:2110.07560) as Lottery Ticket Sparse Fine-Tuning (LT-SFT). Distinct from DiffPruning (AIMech:DiffPruning), which learns continuous L0-relaxed binary masks end-to-end in a single training phase; from FISH Mask (AIMech:FISHMask), which selects sparse masks before training using empirical Fisher information without Phase-1 magnitude-change selection and rewinding; and from MAD-X / AdapterFusion (AIMech:AdapterFusion), which insert serial bottleneck adapter layers that increase inference latency.

Claims & Grounded Evidence

4 Claims

Mechanism #ltsft-two-phase-rewinding-and-sparse-difference-vector

In Phase 1, LT-SFT fully fine-tunes pretrained parameters theta^(0) on dataset D (optionally with L1 regularization J(theta) = (lambda/N) sum_i |theta_i - theta_i^(0)|) to obtain theta^(1) and sets a binary mask mu_i = 1 for the top-K parameters ranked by |theta_i^(1) - theta_i^(0)|; in Phase 2, it resets parameters to theta^(0) without zeroing frozen weights and applies masked gradient updates mu odot nabla_theta L(F(.; theta), D) to obtain theta^(2) and sparse difference vector phi = theta^(2) - theta^(0).

supports Source: Composable Sparse Fine-Tuning for Cross-Lingual Transfer · Locator: Section 3.1 & Algorithm 1, pp. 3-4, 15

Mechanism #ltsft-zero-shot-cross-lingual-functional-composition

For zero-shot cross-lingual transfer from source language s to target language l on task t, LT-SFT learns language SFTs phi_L^(s) and phi_L^(l) via masked language modeling on theta^(0), learns task SFT phi_T^(t) = theta' - (theta^(0) + phi_L^(s)) by running LT-SFT atop source-adapted parameters theta^(0) + phi_L^(s) and subtracting phi_L^(s), and composes them at inference as F_{t,l} = F(.; theta^(0) + phi_T^(t) + phi_L^(l)) with zero additional parameters or latency.

supports Source: Composable Sparse Fine-Tuning for Cross-Lingual Transfer · Locator: Section 3.1-3.2 & Algorithm 1, pp. 4, 15

Application #ltsft-multilingual-pos-dp-ner-nli-and-xquad-transfer

Evaluated across 35 typologically diverse languages on Universal Dependencies POS tagging and dependency parsing, MasakhaNER, and AmericasNLI, LT-SFT outperforms MAD-X across all four tasks (+2.5 POS accuracy, +2.5/+3.7 DP UAS/LAS, +1.8 NER F1, and +1.9 NLI accuracy), and with multi-source training on XLM-R Base surpasses full fine-tuning of XLM-R Large on XQuAD.

supports Source: Composable Sparse Fine-Tuning for Cross-Lingual Transfer · Locator: Section 5, Tables 2-4, pp. 6-8
supports Source: Composable Sparse Fine-Tuning for Cross-Lingual Transfer (Crossref metadata) · Locator: ACL 2022 bibliographic record

Limitation #ltsft-two-phase-training-and-density-interference-limitation

LT-SFT requires two sequential training phases (a full fine-tuning pass followed by a sparse fine-tuning pass), decoupling input and output embeddings while freezing output embeddings and layer normalization during language adaptation, and degrades markedly when mask density exceeds approximately 30% due to parameter overlap and interference.

supports Source: Composable Sparse Fine-Tuning for Cross-Lingual Transfer · Locator: Section 4.2 & Section 5.2, Figure 3, pp. 5, 8-9

⚠️ Curated Limitations & Negative Results

Requires two training phases (Phase 1 full fine-tuning to identify the top-K mask mu followed by Phase 2 masked re-training from theta^(0)), increasing training compute compared to single-phase fine-tuning.
During language MLM adaptation, input and output embedding matrices must be decoupled with the output matrix and layer-normalization parameters held fixed, or else most top-K changes concentrate in the output embeddings and harm downstream transfer.
Cross-lingual composition performance degrades significantly when SFT mask density exceeds ~30% due to parameter overlap and interference between language and task difference vectors.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:SparseLearning` Sparse Learning	narrow_match	LT-SFT is a sparse parameter-adaptation algorithm that learns and composes sparse difference vectors over a fixed pretrained model.
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	LT-SFT adapts pretrained Transformer neural networks across languages and downstream tasks without altering their neural architecture.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r710-primary-pdf`	Composable Sparse Fine-Tuning for Cross-Lingual Transfer Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, Ivan Vulic	primary paper full_text	ID: https://dblp.org/search/publ/api?q=Composable+Sparse+Fine-Tuning+for+Cross-Lingual+Transfer&h=5&format=xml sha256:e3bcda384cf1…
`r710-arxiv-abs`	Composable Sparse Fine-Tuning for Cross-Lingual Transfer (arXiv abstract) Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, Ivan Vulic	primary paper abstract_only	ID: https://arxiv.org/abs/2110.07560 sha256:21d33b154193…
`r710-metadata-cr`	Composable Sparse Fine-Tuning for Cross-Lingual Transfer (Crossref metadata) Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, Ivan Vulic	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=Composable+Sparse+Fine-Tuning+for+Cross-Lingual+Transfer&h=5&format=xml sha256:b73031bc723e…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:13:00Z

Updated: 2026-10-19T13:13:00Z

Source File: data/curated/LTSFTLotteryTicketSparseFineTuning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories