Deep InfoMax
Algorithms AIMech:DeepInfoMax DRAFT
Aliases: DIM
A representation-learning procedure that jointly trains an encoder and a dependence critic on matched versus mismatched global or local-to-global features, optionally regularizing the encoder output toward a chosen prior.
Identity Notes & Concept Scope
Deep InfoMax (DIM) denotes Hjelm et al.’s framework, including explicitly configurable global/local objectives and prior matching. It is not IIC’s discrete cluster-assignment objective or a generic synonym for mutual information. The JSD, Donsker–Varadhan and InfoNCE options have different mathematical and numerical meanings. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 199 preceding native IDs/names/aliases. Ignored-inclusive word-delimited searches of both full checkouts found eight prior discovery/review references in each. Short acronyms were compared in normalized record identities; broad text searches used the full names and longer aliases. Nearest maintained mechanisms and seven exact AIO nodes with incident edges were inspected.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
Encode an input into an intermediate feature map and a global representation. Train encoder and critic together to score same-input feature/representation pairs above pairs formed from different inputs. The global objective relates an image feature map to its global code; the local objective averages dependence scores between local feature locations and the global code. The paper considers a Donsker–Varadhan KL bound, a logistic JSD proxy and an InfoNCE classification objective. For JSD, maximize mean_joint[−softplus(−T)]−mean_product[softplus(T)], up to constants; this is a dependence proxy, not a numerical KL mutual-information value. Combine global and local losses with chosen weights. Optional prior matching trains a separate discriminator to distinguish samples from a chosen prior and encoder outputs, while training the encoder to fool that discriminator. The pinned implementation detaches encoder outputs for discriminator training and uses a non-saturating encoder loss. Its main Fenchel-dual implementation averages same-image scores and masks cross-image scores separately; its InfoNCE branch uses finite negative logits to suppress same-image negatives.
⚠️ Curated Limitations & Negative Results
- A bounded JSD dependence proxy is not identical to KL mutual information. The appendix’s pointwise relationship and sampled correlations do not establish a universal ranking equivalence between changing joint distributions. Finite neural critics and batches further limit estimation.
- Local/global dependence can capture nuisance information. Finite representation capacity, receptive fields, negative sampling and the input distribution affect which shared factors are learned; semantic correctness is not guaranteed.
- The pinned InfoNCE implementation replaces masked negative scores by −10 rather than excluding them with negative infinity. Those terms retain finite mass in its denominator; no exact equality to an ideal exclusion-based InfoNCE formula is asserted.
- Prior matching is an optional adversarial regularizer, not an input reconstruction decoder or a proof of component independence. Finite optimization need not match the requested prior exactly.
- Only declared primary passages and code ranges were inspected. No runtime environment, training, benchmark, mutual-information estimator accuracy or gradient implementation was independently reproduced.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:SelfsupervisedLearning
Self-supervised Learning
|
related_match | This algorithm learns from self-generated augmentation targets or matched local/global input relationships without manual target labels. The target is a task; the relation is not equivalence or an is-a assertion, nor a claim that all experimental preprocessing or selection is label-free. |
Evidence Sources & Primary Literature
5 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio
|
primary paper
full_text
|
ID: https://arxiv.org/abs/1808.06670v5
sha256:8325c5969b7b…
|
additional-1
|
rdevon/DIM contributors
|
official documentation
documentation
|
ID: https://github.com/rdevon/DIM/blob/bac4765a8126746675f517c7bfa1b04b88044d51/cortex_DIM/models/dim.py
sha256:64189b9d75c7…
|
additional-2
|
rdevon/DIM contributors
|
official documentation
documentation
|
ID: https://github.com/rdevon/DIM/blob/bac4765a8126746675f517c7bfa1b04b88044d51/cortex_DIM/functions/dim_losses.py
sha256:5baf43bb648a…
|
additional-3
|
rdevon/DIM contributors
|
official documentation
documentation
|
ID: https://github.com/rdevon/DIM/blob/bac4765a8126746675f517c7bfa1b04b88044d51/cortex_DIM/functions/gan_losses.py
sha256:3de917192425…
|
additional-4
|
rdevon/DIM contributors
|
official documentation
documentation
|
ID: https://github.com/rdevon/DIM/blob/bac4765a8126746675f517c7bfa1b04b88044d51/cortex_DIM/models/prior_matching.py
sha256:86b990d4cfdc…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T08:30:19Z
Updated: 2026-10-11T08:31:31Z
Source File:
data/curated/DeepInfoMax.yaml