Home / Curated / Layers & Modules / Hierarchical softmax

Hierarchical softmax

Layers & Modules AIMech:HierarchicalSoftmax DRAFT

A neural categorical-output component that assigns a probability to each vocabulary leaf by multiplying normalized branch probabilities along its path through a word tree.

Identity Notes & Concept Scope This record covers the hierarchical probability output component, including the binary sigmoid form described as hierarchical softmax by Mikolov and colleagues. Morin and Bengio’s parameter-sharing language model and Mnih and Hinton’s hierarchical log-bilinear model are complete architectures using this factorization, not aliases for the component. A Huffman tree, a WordNet-derived tree and a learned tree are construction choices with different scope. All 442 pinned AIO labels/scoped synonyms and 352 preceding native IDs/names/aliases were compared, with no normalized identity match. Both full-checkout searches included ignored and hidden files; worktree returned 15 lines and main 19. All 16 unique matched contents were read: DocNADE contrasts, frontier and review references, plus four ignored-environment Sage/Pygments entries unrelated to AI SAGE. Twenty-one exact AIO nodes and four complete native neighbors were inspected. Absence is bounded to this snapshot and semantic neighborhood.

Claims & Grounded Evidence

1 Claim

Mechanism #method

In a binary tree with one leaf per word, each internal node computes a branch probability from the input representation, commonly sigmoid of a node-vector dot product with optional bias. The opposite branch has its complement. Multiply the appropriate branch probabilities on the target word’s path to obtain its exact probability under this tree-parameterized distribution; probabilities over all leaves sum to one. The negative log probability is a sum of binary losses, so only nodes on the observed path contribute output-parameter gradients. For a sigmoid node logit and branch indicator y, that loss derivative is sigmoid(logit)-y. Computing one target probability and its gradient costs work proportional to its path length, in addition to representation computation. The source’s skip-gram variant uses internal-node output vectors and frequency-based Huffman codes. Multiple leaves assigned to one word require summing their disjoint path probabilities, as in the HLBL extension.

supports Source: Distributed Representations of Words and Phrases and their Compositionality · Locator: Section 2.1, Equation (3) and internal-node/Huffman discussion, PDF/printed p.3; skip-gram objective context, Section 2, p.2
supports Source: Hierarchical Probabilistic Neural Network Language Model · Locator: Sections 3–4, path factorization, shared node predictor and full-vocabulary cost distinction, PDF pp.3–4; hierarchy choices, Section 5, pp.4–5
supports Source: A Scalable Hierarchical Distributed Language Model · Locator: Section 4, Equations (3)–(5), PDF/printed pp.3–4; tree construction alternatives, Section 5, pp.4–5

⚠️ Curated Limitations & Negative Results

The tree parameterization defines a normalized distribution of its own. Calling it an approximation to full softmax does not assert equality to probabilities from a particular flat-softmax parameter set or provide a numerical approximation-error bound.
Logarithmic target cost requires suitable depth or weighted-average code length. A full binary tree alone can have linear worst-case depth. Even a Huffman code’s mean length need not be at most log2(vocabulary size): three equally frequent words have optimal mean length 5/3, exceeding log2(3). Thus the unqualified average-length sentence in Mikolov Section 2.1 is not adopted as a universal bound.
Computing every vocabulary probability requires at least linear output work. The short-path calculation does not by itself give a logarithmic-time exact top-k or most-probable-word search.
Shared internal nodes, tree construction and input representation constrain the fitted model. Multiple codes per word add paths and require their sum, rather than selecting an arbitrary single leaf or averaging leaf probabilities.
This component is not a whole language model, a hard decision tree, negative sampling, or Gumbel-Softmax sampling. No empirical speedup, embedding-quality ranking or author-code behavior is certified.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:OutputLayer` Output Layer	broad_match	The primary hierarchical-softmax description explicitly represents the neural output through a word tree. OutputLayer is broader than this particular probability-producing component. The SoftmaxLayer target specifically applies the flat softmax function to its inputs and is not asserted equivalent or broader merely because the names overlap. HierarchicalLanguageModel is a complete model of multilevel representations, while this component does not specify its surrounding language architecture.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`method-source`	Distributed Representations of Words and Phrases and their Compositionality Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean	primary paper full_text	ID: https://proceedings.neurips.cc/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html sha256:9ee42f08c5e6…
`original-source`	Hierarchical Probabilistic Neural Network Language Model Frederic Morin, Yoshua Bengio	primary paper full_text	ID: pmlr-vR5-morin05a sha256:43409245b14c…
`hlbl-source`	A Scalable Hierarchical Distributed Language Model Andriy Mnih, Geoffrey Hinton	primary paper full_text	ID: https://proceedings.neurips.cc/paper/2008/hash/1e056d2b0ebd5c878c550da6ac5d3724-Abstract.html sha256:1532535af39d…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-11T18:02:59Z

Updated: 2026-10-11T18:02:59Z

Source File: data/curated/HierarchicalSoftmax.yaml