Infomax Independent Component Analysis
Algorithms AIMech:InfomaxICA DRAFT
Aliases: Infomax ICA, Bell-Sejnowski ICA
An unsupervised source-separation learning algorithm that adjusts an invertible square linear transform and biases by the ordinary gradient of the differential entropy of coordinatewise sigmoid-transformed outputs.
Identity Notes & Concept Scope
Scoped to the standard logistic instantaneous algorithm of Bell and Sejnowski 1995. The unsquashed linear outputs estimate sources; the sigmoid outputs define its entropy contrast. This is distinct from the broader Infomax principle, Deep InfoMax’s learned dependence critics, later natural-gradient ICA, FastICA, and the paper’s separately developed temporal deconvolution and generalized-sigmoid variants. Four candidate identities were compared against all 442 pinned AIO IDs, labels and scoped synonyms and 450 preceding native IDs, names and aliases, with no normalized match. Ignored/hidden full-checkout searches returned the same 106 matched-line set in each checkout after the prior merge. All matched excerpts from 76 files were read; they are discovery context, not existing concept records. The semantic search covered all pinned labels and scoped synonyms and returned 212 rows: 205 byte-equivalent prior readings and seven additional rows, all inspected. Twenty-three annotated AIO nodes with outgoing edges comprise 21 unchanged prior readings and two new inspections. Thirty contextual YAMLs, including seven native neighbors, comprise 25 unchanged prior readings and five complete new readings. This absence assessment is bounded to the pinned snapshot and inspected maintained corpus.
Claims & Grounded Evidence
3 Claims
Mechanism
#entropy-gradient
Let u=Wx+b with square invertible W and y_i=1/(1+exp(-u_i)). The output differential entropy differs from H(X) by log|det W|+sum_i E[log(y_i(1-y_i))]. Its ordinary gradient updates are proportional to W^(-T)+(1-2y)x^T for W and 1-2y for b, averaged or sampled over observations. This is the ordinary parameter gradient, not a natural-gradient replacement. The inverse term is undefined at singular W, and finite steps need not preserve invertibility automatically.
Limitation
#information-and-independence
The curated objective is output differential entropy. For noiseless invertible transformations of continuous variables, mutual information between input and output is infinite, so the finite change-of-variables contrast is not a finite noiseless mutual-information value. A finite added output-noise model has a different noisy-output entropy unless the appropriate limiting approximation is justified. Maximizing the sigmoid-output entropy also trades off marginal entropy and dependence: a fixed nonlinearity mismatched to source distributions can prefer mixed outputs. The source illustrates this for uniform sources; arbitrary stationary points do not certify independent components.
Scope
#source-separation-scope
The instantaneous application assumes a linear mixture x=A s and seeks unsquashed outputs u that recover sources up to scale, sign and permutation under suitable distributional and identifiability conditions. The historical demonstrations mix recorded audio and discuss failures with multiple Gaussian white-noise sources and ill-conditioned mixing. The paper separately derives FIR blind deconvolution, which can whiten a nonwhite source instead of recovering the exact original. Neither arbitrary acoustic propagation delays nor unmodeled input noise is handled by the standard instantaneous rule.
⚠️ Curated Limitations & Negative Results
- Square nonsingular mixing, suitable source distributions and an appropriate contrast are substantive assumptions. The standard fixed logistic rule is not a universal source-separation or independence guarantee.
- The generalized sigmoid ODE in Equation 2.32 gives a chain-rule factor y^(p-1)(1-y)^(r-1) multiplying p(1-y)-r*y; Equations 2.33-2.34 omit it outside the standard p=r=1 case. Curator arithmetic verifies this discrepancy. Those generalized variants are excluded here; no uninspected author-code defect is asserted.
- Gradient ascent with a chosen finite learning rate need not converge globally or monotonically. The inverse-weight term is nonlocal; the source does not establish a biological learning mechanism.
- Audio demonstrations, annealing schedules and comparative timings are historical source-specific evidence. No author implementation, audio separation experiment or benchmark was run.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:UnsupervisedLearning
Unsupervised Learning
|
related_match | The algorithm estimates source-separating parameters from observations without labeled source targets. The task-scoped AIO target is related to this algorithm without equating their abstraction levels. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Anthony J. Bell, Terrence J. Sejnowski
|
primary paper
full_text
|
ID: doi:10.1162/neco.1995.7.6.1129
sha256:abcb3468dc3c…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-12T01:49:01Z
Updated: 2026-10-12T01:49:01Z
Source File:
data/curated/InfomaxICA.yaml