Lightweight and Dynamic Convolutions (LightConv and DynamicConv)
Layers & Modules AIMech:PayLessAttentionWithLightweightAndDynamicConvolutions DRAFT
Aliases: LightConv and DynamicConv, Pay Less Attention with Lightweight and Dynamic Convolutions, Softmax-Normalized Lightweight and Dynamic Sequence Convolutions
Lightweight Convolution (LightConv) and Dynamic Convolution (DynamicConv) are linear-time O(n k d) sequence modeling primitives that replace quadratic O(n^2 d) content-based self-attention: LightConv applies a depthwise separable 1D convolution (preceded by a d -> 2d linear projection and Gated Linear Unit) whose kernel weights W in R^{H x k} are tied across every group of d/H contiguous channels, normalized across the temporal dimension k via softmax(W)_{h,j} = exp(W_{h,j}) / sum_{j'=1}^k exp(W_{h,j'}), and regularized with DropConnect; DynamicConv extends LightConv by predicting time-step-dependent convolution kernels f(X_i) = W^Q X_i in R^{H x k} as a learned linear function of the current time-step input X_i alone.
Identity Notes & Concept Scope
Distinct from standard 1D or depthwise separable convolutions (Kaiser et al., 2018), which neither tie weights across d/H channel groups nor apply temporal softmax normalization or per-timestep kernel prediction, and distinct from Lite Transformer (AIMech:LiteTransformerWithLongShortRangeAttention), which combines a self-attention global branch and a convolution local branch in parallel rather than replacing self-attention across all encoder and decoder layers.
Claims & Grounded Evidence
4 Claims
Mechanism
#lightconv-channel-group-weight-sharing-and-temporal-softmax
LightConv ties depthwise convolution parameters across every group of d/H contiguous channels—reducing kernel parameters from d^2 * k (standard convolution) and d * k (depthwise convolution) to H * k (e.g., from 7,340,032 to 112 parameters for d = 1024, k = 7, H = 16)—and normalizes weights across the temporal window k via softmax(W)_{h,j} = exp(W_{h,j}) / sum_{j'=1}^k exp(W_{h,j'}) so that LightConv(X, W_{ceil(cH/d),:}, i, c) = DepthwiseConv(X, softmax(W_{ceil(cH/d),:}), i, c).
Mechanism
#dynamicconv-timestep-dependent-linear-kernel-prediction
DynamicConv predicts a separate H x k convolution kernel at each time step i solely from the current token representation X_i via a learned linear module f(X_i)_{h,j} = sum_{c=1}^d W^Q_{h,j,c} X_{i,c} and computes DynamicConv(X, i, c) = LightConv(X, f(X_i)_{h,:}, i, c), achieving position-varying context weights with O(n) complexity in sequence length n.
Application
#dynamicconv-translation-summarization-and-runtime-speedup
Without using self-attention in either encoder or decoder, DynamicConv achieves 29.7 BLEU on WMT'14 En-De (+0.4 BLEU over Transformer-Big), 43.2 BLEU on WMT'14 En-Fr, +0.8 BLEU on IWSLT'14 De-En, 20% faster inference throughput than self-attention, 26.67 test perplexity on Billion Word language modeling, and higher ROUGE on CNN-DailyMail summarization.
Limitation
#dynamicconv-softmax-normalization-stability-and-receptive-field-limitation
Removing temporal softmax normalization (or replacing it with unnormalized, L1-normalized, squared, or abs weights) causes DynamicConv training to diverge (Table 3 and Appendix A Table 6), and each layer's receptive field is bounded by its fixed kernel width k rather than attending globally in a single hop.
⚠️ Curated Limitations & Negative Results
- DynamicConv training diverges without temporal softmax (or L2/sigmoid/tanh) kernel normalization.
- Each layer has a finite receptive field bounded by kernel width k, requiring progressive multi-layer kernel expansion (e.g., k = 3, 7, 15, 31) to capture long-range context.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
broad_match | LightConv and DynamicConv are neural sequence modeling layer primitives and encoder-decoder architectures for natural language processing. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r820-primary-pdf
|
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, Michael Auli
|
primary paper
full_text
|
ID: https://arxiv.org/bibtex/1901.10430
sha256:9d3da8563d94…
|
r820-arxiv-abs
|
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, Michael Auli
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1901.10430
sha256:0d313838f4bf…
|
r820-metadata-cr
|
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, Michael Auli
|
primary paper
abstract_only
|
ID: https://arxiv.org/bibtex/1901.10430
sha256:6adaac1f23ca…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T15:03:00Z
Updated: 2026-10-19T15:03:00Z
Source File:
data/curated/PayLessAttentionWithLightweightAndDynamicConvolutions.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories