Home / Curated / Algorithms / Vote-k Selective Annotation for In-Context Learning

Vote-k Selective Annotation for In-Context Learning

Algorithms AIMech:VoteKSelectiveAnnotationForInContextLearning DRAFT

Aliases: Vote-k Selective Annotation, Vote-k Graph-Based Selective Annotation, Selective Annotation with Vote-k

A graph-based selective annotation algorithm for few-shot in-context learning that constructs a directed k-nearest-neighbor graph over dense sentence embeddings of an unlabeled dataset, iteratively selects an initial fraction (M/10) of examples to annotate by maximizing a discounted in-degree score where each vertex's vote is exponentially decayed by the number of its already-selected neighbors, evaluates the language model's predictive confidence on the remaining unlabeled pool using those initial annotations, and selects the remaining 9M/10 examples via the discounted vote-k score stratified across confidence-percentile buckets.

Identity Notes & Concept Scope Distinct from test-time prompt retrieval methods such as KATE (AIMech:KATEkNNAugmentedInContextExampleSelection) and EPR (AIMech:EPREfficientPromptRetrievalForInContextLearning), which assume the entire training pool is already annotated and select prompts per test query, whereas Vote-k operates before annotation to choose a compact, diverse, and representative subset L of size M from an unlabeled pool X to be human-annotated. Also distinct from pure facility-location or greedy embedding-diversity coreset selection, which do not combine exponentially discounted kNN graph voting with in-context LM confidence stratification.

Claims & Grounded Evidence

4 Claims

Mechanism #votek-discounted-knn-graph-scoring-mechanism

Vote-k builds a directed graph G = (V, E) connecting each unlabeled instance to its k nearest cosine-similarity neighbors in Sentence-BERT embedding space and scores each candidate u by summing s(v) = rho^{-|{l in L | (v, l) in E}|} (rho > 1) over all remaining vertices v that have an edge to u, promoting both high neighborhood density and coverage diversity.

supports Source: Selective Annotation Makes Language Models Better Few-Shot Learners · Locator: Section 2.1 & Appendix G.1 Algorithm 1 (Lines 2-7), pp. 3 & 21

Mechanism #votek-confidence-bucket-stratification-mechanism

After annotating the first M/10 examples selected by Vote-k, the algorithm uses them as in-context demonstrations to compute average log-probability confidence scores on all remaining unlabeled instances, partitions the unlabeled pool into M equal-sized confidence percentile buckets, and selects the highest-scoring Vote-k instance from each of the first 9M/10 buckets while discarding the top 10% most confident buckets.

supports Source: Selective Annotation Makes Language Models Better Few-Shot Learners · Locator: Section 2.1 & Appendix G.1 Algorithm 1 (Lines 8-18), pp. 3 & 21

Application #votek-annotation-efficiency-and-domain-shift-application

Paired with similarity-based prompt retrieval across 10 classification, commonsense reasoning, dialogue state tracking, semantic parsing, QA, and summarization benchmarks, Vote-k achieves a 12.9% and 11.4% relative gain over random annotation under budgets of 18 and 100 examples and matches supervised fine-tuning with 10-100x fewer annotations.

supports Source: Selective Annotation Makes Language Models Better Few-Shot Learners · Locator: Section 3.3 & Section 4.1-4.3, pp. 5-8, Tables 2-3 & Figure 2
supports Source: Selective Annotation Makes Language Models Better Few-Shot Learners (Crossref metadata) · Locator: DBLP bibliographic record Hongjin Su et al. ICLR 2023

Limitation #votek-similarity-retrieval-coupling-limitation

The diversity and representativeness gains of Vote-k selective annotation depend critically on pairing the annotated pool with similarity-based test-time prompt retrieval; when prompts are sampled randomly at test time from the Vote-k pool, performance drops and fails to improve over random annotation.

supports Source: Selective Annotation Makes Language Models Better Few-Shot Learners · Locator: Section 4.4, p. 8, Table 4

⚠️ Curated Limitations & Negative Results

The full Vote-k algorithm requires running language model inference over the remaining unlabeled pool after the initial M/10 annotations to compute confidence percentiles (though Fast Vote-k omits this step).
Directly fine-tuning a language model on the Vote-k selected subset without outlier removal does not yield the same gains observed in similarity-retrieved in-context learning.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Vote-k performs graph-based selective annotation over neural sentence embeddings and language model confidence scores to construct few-shot demonstration pools.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r738-primary-pdf`	Selective Annotation Makes Language Models Better Few-Shot Learners Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, Tao Yu	primary paper full_text	ID: https://dblp.org/search/publ/api?q=Selective+Annotation+Makes+Language+Models+Better+Few-Shot+Learners+Su&h=5&format=xml sha256:f9dd0373bace…
`r738-arxiv-abs`	Selective Annotation Makes Language Models Better Few-Shot Learners (arXiv abstract) Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, Tao Yu	primary paper abstract_only	ID: https://arxiv.org/abs/2209.01975 sha256:79f954928366…
`r738-metadata-cr`	Selective Annotation Makes Language Models Better Few-Shot Learners (Crossref metadata) Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, Tao Yu	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=Selective+Annotation+Makes+Language+Models+Better+Few-Shot+Learners+Su&h=5&format=xml sha256:a1fa4bc6c510…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:41:00Z

Updated: 2026-10-19T13:41:00Z

Source File: data/curated/VoteKSelectiveAnnotationForInContextLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories