Non-local block
Layers & Modules AIMech:NonLocalBlock DRAFT
Aliases: Non-local neural-network block
A neural-network component that aggregates input-dependent pairwise contributions from positions across a representation, projects the aggregate and adds it to the original features through a residual connection.
Identity Notes & Concept Scope
This record denotes the residual non-local block introduced in the 2018 paper, not an entire non-local network, classical non-local-means denoiser, fixed dense layer or a particular trained video classifier. Its operator family includes softmax and non-softmax pairwise functions; an embedded-Gaussian instance overlaps self-attention without making every family member identical to a Transformer layer. No normalized identity match was found across all 442 pinned AIO labels/scoped synonyms and 282 preceding native IDs/names/aliases. Full-checkout fixed-string searches included ignored and hidden files in worktree and main, returning 2 and 2 lines respectively. Every text hit was a prior candidate lead or a closely related variant lead. Ten exact AIO CLASS nodes with scoped synonyms/comments and outgoing edges and five native neighboring mechanisms were inspected. Absence is bounded to this snapshot and semantic neighborhood.
Claims & Grounded Evidence
1 Claim
Mechanism
#method
At each output position, evaluate a pairwise scalar function between that position and source positions across space, time or spacetime. Use those scalars to aggregate learned value embeddings of source features, with a normalizer chosen for the pairwise function. The paper’s Gaussian and embedded-Gaussian variants exponentiate dot products and normalize each query over source positions, yielding softmax weights. Its raw embedded dot-product and ReLU-on-concatenated-embeddings variants divide by the number of source positions instead. Project the aggregate back to the input channel count and add the original feature at the output position. A zero residual branch can initialize the block as identity. The source reduces embedding width and optionally max-pools the projected source branches to reduce pairwise work while retaining the output query grid; this changes the sampled context but retains long-range aggregation.
⚠️ Curated Limitations & Negative Results
- Non-softmax weights need not sum to one, and raw dot-product weights can be negative. Their aggregate need not be a convex average or calibrated attention distribution. The paper’s Gaussian terminology refers to its exponential dot-product formula, not a required Euclidean radial-basis kernel.
- The embedded-Gaussian formula lacks an explicit inverse-square-root dimension factor and is not a complete multi-head Transformer architecture. Subsampling projected source branches does not generally reproduce the full-resolution all-pairs result or commute with projection and pooling.
- Zero output projection in the stated residual formula gives identity only when all residual contributions vanish. In the source’s batch-normalized implementation, zero scale also needs zero offset to make the branch zero; later training changes this state.
- Explicit pairwise computation can require a query-by-source matrix and becomes expensive at high resolution. Bottleneck width, subsampling, placement and the chosen implementation determine costs; global reach does not imply linear complexity or universal speedup. No benchmark replication, source-code execution or proof of causal interpretability is asserted.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:Layer
Layer
|
broad_match | The native concept is a defined representation-transforming component within a neural network, so the generic Layer class has broader scope. |
AIO:AttentionLayer
Attention Layer
|
related_match | The embedded-Gaussian instance computes a softmax of pairwise embedded dot products and is directly related to the pinned dot-product attention layer. The native record also covers non-softmax functions and a residual wrapper, so the relationship is not family-wide equivalence or a forced subclass. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
method-source
|
Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He
|
primary paper
full_text
|
sha256:c593aef4c3bc…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-11T12:54:15Z
Updated: 2026-10-11T12:54:15Z
Source File:
data/curated/NonLocalBlock.yaml