Sparse additive generative model
Model Families AIMech:SparseAdditiveGenerativeModel DRAFT
Aliases: SAGE
A generative model family for categorical text data that forms word probabilities by normalizing a background log-frequency vector plus sparse component deviations, allowing multiple generative facets to combine additively in log space.
Identity Notes & Concept Scope
SAGE here denotes Eisenstein, Ahmed and Xing’s sparse additive generative construction, not Sage mathematical software or graph-neural aggregation. Its classification, topic and multifaceted uses share this parameterization; it is not synonymous with all naive Bayes models, sparse dictionary learning, sparse topic proportions or an arithmetic mixture of component word distributions. All 442 pinned AIO labels/scoped synonyms and 352 preceding native IDs/names/aliases were compared, with no normalized identity match. Both full-checkout searches included ignored and hidden files; worktree returned 15 lines and main 19. All 16 unique matched contents were read: DocNADE contrasts, frontier and review references, plus four ignored-environment Sage/Pygments entries unrelated to AI SAGE. Twenty-one exact AIO nodes and four complete native neighbors were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
For a component k, set p(w=i|k) proportional to exp(m_i+eta_ki), normalizing over the vocabulary. Multiple selected facets contribute a sum of their deviation vectors before this normalization; no additional token-level switch is needed to choose which facet generates the word. This does not remove latent topic assignments used by a topic-model variant. The paper places sparsity-inducing priors on deviations, using zero-mean Gaussian components with latent variances: an exponential variance prior gives a Laplace marginal, while an alternative improper Jeffreys prior gives a different shrinkage construction. It combines MAP estimates of deviations with variational treatment of other latent quantities. For a single component and fixed expected precisions, the gradient is observed or expected counts minus total count times word probabilities minus the precision-weighted deviations; its Hessian is a rank-one term minus a diagonal term, enabling the reported efficient Newton update.
⚠️ Curated Limitations & Negative Results
- Sparsity concerns log deviations. Finite logits still give positive word probabilities, and even zero-deviation words can change absolute probability through normalization; their relative odds remain those of the background when their deviations agree.
- Additive log facets yield multiplicative unnormalized effects, not an arithmetic probability mixture. The model family does not require every document to have a single observed class or prohibit latent variables.
- The Jeffreys-prior paragraph on PDF p.4 prints expected inverse variance equal to eta squared. Its preceding Gamma moment and b=eta squared/(a-1) instead give 1/eta squared for nonzero eta, with a>1 and positive scale. The printed final identity is not adopted; zero deviations require a limiting/numerical treatment. The improper prior is not represented as a normalized marginal distribution.
- The paper warns that warm-starting tightly coupled deviation/variance updates can lock in early sparsity patterns. Neither approximate inference nor Newton updates establish global convergence or calibrated posterior uncertainty.
- The single-component rank-one Hessian argument is not asserted unchanged for sums over multiple facet combinations. No author implementation, universal predictive advantage or numerical benchmark result is certified.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ProbabilisticTopicModel
Probabilistic Topic Model
|
related_match | SAGE is explicitly used to construct probabilistic topic models, but its family also includes observed-class and multifaceted generative text models. The AIO topic-model task therefore provides a related application/modeling context, not an equivalent identity or a necessary superclass. SparseLearning denotes sparse dictionary representations, SparseAutoEncoder denotes hidden-unit constraints, and SparseLLM denotes language-model compression, none of which identifies sparse log-frequency deviations. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Jacob Eisenstein, Amr Ahmed, Eric P. Xing
|
primary paper
full_text
|
ID: Eisenstein_Ahmed_Xing_ICML11
sha256:214c601b2008…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T18:02:59Z
Updated: 2026-10-11T18:02:59Z
Source File:
data/curated/SparseAdditiveGenerativeModel.yaml