Entropy Search
Algorithms AIMech:EntropySearch DRAFT
A Bayesian global optimization algorithm that approximates the location of a function minimum on sampled representer points using expectation propagation, and chooses evaluations by an approximate expected reduction in relative-entropy loss for that minimizer belief.
Identity Notes & Concept Scope
This record maintains the original ES representer-point, expectation-propagation and prospective relative-loss algorithm. It shares an ideal fixed-hyperparameter optimizer-information target with PES after consistent min/max conversion, while its operational approximations differ. Existing FABOLAS uses an ES-related acquisition within a dataset-size/cost procedure and does not cover this original identity. GP regression, cross-entropy elite refitting and generic active learning are distinct. No acronym alias is adopted; actual current identity and ignored-inclusive searches are refreshed before creation.
Claims & Grounded Evidence
6 Claims
Mechanism
#objective-surrogate-and-minimizer-belief
The source models an unknown objective on a bounded continuous domain with a Gaussian process and a specified observation likelihood; Gaussian observation noise gives the analytically tractable core. It seeks a belief over the location of the minimum under regularity assumptions that make this optimizer law meaningful. The evaluation policy aims to improve that belief, rather than always evaluating a minimizer of the posterior mean. This probabilistic target is different from observing the true optimizer or certifying a global optimum.
Mechanism
#nonuniform-representer-point-approximation
A finite set of representer points approximates the continuous minimizer law. The sampling proposal is required to assign nonzero mass to every open neighborhood of the domain; the source discusses probability-of-improvement and expected-improvement shaped proposals as heuristics. These proposal values control the discretization and its density correction. They are neither the minimizer probabilities themselves nor the final information acquisition. Numerical optimization of the next evaluation location need not select one of the representer points.
Mechanism
#expectation-propagation-minimum-probabilities
Restricting the Gaussian process to the representer points gives a multivariate Gaussian over their function values. In the usual no-tie finite Gaussian setting, for each point the event that its value is no greater than the other represented values is a set of pairwise inequalities. Expectation propagation approximates the corresponding Gaussian integrals and provides derivatives of the resulting finite minimizer-probability approximation with respect to the Gaussian mean and covariance. Finite representation and expectation propagation are separate approximations; neither constitutes exact inference of the continuous optimizer law.
Mechanism
#relative-entropy-loss-and-information-target
The source uses negative Kullback–Leibler divergence from the minimizer belief to a fixed uniform base as its loss. On the nonuniform discretization, the varying part of the approximate loss is H(q)-sum_i q_i log u_i, where q is the finite minimizer-probability representation and u_i are the unnormalized proposal values. The omitted terms are constant for that fixed discretization and uniform base. Entropy of the categorical weights alone therefore omits the proposal correction. With exact Bayesian updating, fixed hyperparameters and well-defined finite entropy quantities, the ideal expected loss reduction is information between optimizer location and a future noisy observation. It is the same ideal information objective rearranged by PES, after consistently changing minimum to maximum. This identity does not equate their inference approximations.
Mechanism
#approximate-prospective-loss-and-greedy-query
Prospective Gaussian-process mean and covariance changes are combined with derivatives from the minimizer-probability approximation to approximate the expected change of loss. The source uses an expansion and Monte Carlo integration over observation innovations, keeping the sampled innovations fixed while numerically optimizing the candidate location. After evaluating the selected location, the algorithm updates its data and repeats. This is a greedy one-step policy, rather than optimization of all remaining evaluations jointly. This claim describes the method; it does not adopt the unresolved literal innovation and Taylor notation as a verified implementation.
Scope
#hyperparameter-and-final-decision-scope
The paper discusses learning kernel and likelihood hyperparameters by maximum likelihood or Bayesian sampling. To reuse the Gaussian-process calculation after Bayesian hyperparameter learning, it proposes approximating a mixture of Gaussian processes by a single process using expected mean and covariance functions. It leaves the full hierarchical learning process in the evaluation decision outside its scope. At its horizon, Algorithm 1 returns the approximate minimizer belief for a final decision. The method is therefore not defined by a claim that hyperparameters can never be learned, or by a guaranteed final optimum.
⚠️ Curated Limitations & Negative Results
- This record denotes the original representer-point and expectation propagation algorithm, not all information-based Bayesian optimization. Predictive Entropy Search uses different optimizer sampling and predictive-entropy approximations. Shared ideal information does not imply identical acquisitions on every problem or justify a claim that their selected points always differ.
- Figure 2 on PDF p. 7 explicitly allows atoms at a domain boundary. An optimizer law with such atoms is not a Lebesgue density, and its KL divergence to a purely continuous uniform base is infinite. The ideal density-entropy calculation requires appropriate measures and finite quantities; the finite discretized loss is an approximation. Appendix A does not provide a general existence or convergence theorem for every objective prior. Degenerate priors, singular finite covariances and tied optimizer values require explicit handling beyond the ordinary no-tie Gaussian calculation.
- Section 2.4 on PDF p. 11 and Equation 6 on PDF p. 13 leave literal signed covariance innovation, noise placement and second-mean-derivative coefficient questions for implementation audit. This record neither silently repairs those expressions nor claims to reproduce them numerically.
- The expected-improvement comparator on PDF p. 4 prints min{0,eta-f} although its closed form is the nonnegative improvement expression. The optional logarithmic-likelihood discussion on PDF p. 17 calls logarithm convex and prints a positive second derivative where the stated likelihood gives -y^2/sigma^2 at g=log(y). Those contextual discrepancies are not adopted as core Entropy Search formulas.
- The retained curator calculator checks 27 small positive binary channels and separate constructed boundary examples. Probability masses use exact rational arithmetic; entropy uses floating logarithms. These checks illustrate information identities and limitations. They do not run Gaussian-process fitting, expectation propagation, acquisition optimization or author code, and establish no general accuracy, runtime, regret or global convergence guarantee.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ActiveLearning
Active Learning
|
related_match | The pinned class describes interactive querying of an information source for outputs. This algorithm chooses informative objective evaluations to update its surrogate and optimizer belief. The relation records that query-learning association. It does not equate black-box optimization with annotation or pool-based classification, assert subclass membership, or identify the specific acquisition with the broad AIO class. |
Evidence Sources & Primary Literature
2 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
hennig2012
|
Philipp Hennig, Christian J. Schuler
|
primary paper
full_text
|
sha256:c1141315e0ce…
|
hernandez2014
|
José Miguel Hernández-Lobato, Matthew W. Hoffman, Zoubin Ghahramani
|
primary paper
full_text
|
sha256:82eb9e614937…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-15T00:57:14Z
Updated: 2026-10-15T00:57:14Z
Source File:
data/curated/EntropySearch.yaml