Topic-model evocation score
Evaluation Methods AIMech:TopicModelEvocationScore DRAFT
Aliases: Top-m evocation metric
A topic-model evaluation method that orders judged word pairs by model-derived weights and sums their human evocation ratings over a chosen number of highest-ranked pairs, with alternative aggregations across topics.
Identity Notes & Concept Scope
This record denotes the top-m scoring procedure and its Evoc-1/Evoc-2 aggregation choices, not the human evocation dataset, a word-similarity embedding, a distance-metric learning algorithm or an empirical score achieved by a trained model. Human evocation measures how strongly one term brings another to mind; it need not mean synonymy. The two aggregations are retained as explicitly different operations within the scoring method rather than asserted as synonymous numerical quantities. No normalized identity match was found across all 442 pinned AIO labels and scoped synonyms and 421 preceding native IDs, names and aliases. Full-checkout fixed-string searches in worktree and main included ignored and hidden files. Ten matched lines in each checkout concern prior discovery references, raw search/abstract captures and downloaded-source metadata; these were classified by their file/field context and matched excerpts, not treated as ontology identities. All 192 semantic label/synonym hits were read. Nineteen exact annotated CLASS nodes and outgoing edges were inspected, with 13 verified equal to prior reading and six newly read. Thirty-four contextual YAMLs comprise 19 AIO and 15 native records; eight were fully read here, three final round99 records were hash-verified after full reading, and 23 were byte-verified after earlier full reading. Absence is bounded to the pinned snapshot, maintained corpus and inspected semantic neighborhood.
Claims & Grounded Evidence
2 Claims
Mechanism
#score-and-aggregation
Let E be a finite set of word pairs with human scores h_e and model weights M_e, and choose an integer m with 0<=m<=|E|. Order E by descending M using an explicit tie convention, and set Evoc_m(M,h)=sum_{r=1}^m h_{pi_M(r)}. For K topics, Evoc-1 is the unweighted mean of the K separate top-m scores; Evoc-2 is the top-m score after averaging their weight vectors over the same judged pairs. The paper derives APM weights mainly from edge coefficients, with node-based fallback for insufficient nonzero edges, and uses products of topic-word probabilities for independent multinomial topic models. A strictly increasing transformation of one weight vector preserves its rank-based score under the same tie handling. Separately rescaling topics before their weight vectors are averaged can change Evoc-2 even when every individual score and Evoc-1 stays unchanged.
Scope
#evaluation-scope
The reported evaluation maps human-rated synsets to word/POS entries, averages two directed ratings to obtain undirected pair scores, and retains 1646 such entries across 4049 BNC documents. Human-scored pairs are split equally between tuning and testing for model selection; this is distinct from holding out all test documents from unsupervised fitting. The reported comparisons use m=50, selected topic counts and model-specific parameter grids. Those design choices specify this historical evaluation and do not establish universal topic coherence, causal relations or performance on unjudged word pairs.
⚠️ Curated Limitations & Negative Results
- The score depends on the judged-pair subset, rating scale, m and tie convention. Unjudged pairs are excluded rather than automatically assigned zero. A larger sum is not a calibrated probability, a distance satisfying metric axioms or an estimate of all possible word-pair quality.
- Evoc-1 and Evoc-2 generally differ. Individual rank invariance does not justify independently changing topic scales before computing Evoc-2. The source does not fully specify every fallback and tie-handling detail, so numerical replication requires additional implementation conventions.
- Human evocation can be directional, whereas the historical evaluation symmetrizes it and collapses word senses to word/POS. These choices discard information and need not fit every application or language.
- Human annotation and pair selection can bias coverage. Tuning on one half of pair judgments and evaluating on the other does not establish an unseen-document or unseen-vocabulary test. Topic labels, selected word-pair examples and authors superiority claims do not guarantee semantic validity.
- No evocation dataset was analyzed and no author evaluation script or benchmark was executed. The mathematical witnesses are synthetic checks of ranking and aggregation only.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ProbabilisticTopicModel
Probabilistic Topic Model
|
related_match | This evaluation procedure directly assesses word-pair structure represented by topic models. The task-scoped AIO class is related to the assessed object, not a broader evaluation criterion or an equivalent score. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
David I. Inouye, Pradeep Ravikumar, Inderjit S. Dhillon
|
primary paper
full_text
|
sha256:f38e28540906…
|
method-supplement
|
David I. Inouye, Pradeep Ravikumar, Inderjit S. Dhillon
|
primary paper
full_text
|
sha256:97c115b77a70…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T23:32:13Z
Updated: 2026-10-11T23:32:13Z
Source File:
data/curated/TopicModelEvocationScore.yaml