Lite Transformer with Long-Short Range Attention (LSRA)
Architectures AIMech:LiteTransformerWithLongShortRangeAttention DRAFT
Aliases: Lite Transformer, Long-Short Range Attention, LSRA Lite Transformer
Lite Transformer with Long-Short Range Attention (LSRA) is a computation- and memory-efficient Transformer architecture for resource-constrained mobile NLP (<500M Mult-Adds, <10M parameters) that eliminates the standard 1:4 FFN channel expansion bottleneck by flattening the attention and FFN channel dimensions and replaces standard multi-head attention with a parallel two-branch LSRA primitive: the input channels are split equally into two d/2 halves, where the global extractor branch applies multi-head self-attention to capture sparse long-range relationships and the local extractor branch applies gated linear units (GLU) and lightweight depthwise separable 1D convolutions to capture diagonal local neighborhood context before channel concatenation and FFN mixing.
Identity Notes & Concept Scope
Distinct from Evolved Transformer (So et al., 2019), which stacks convolution and self-attention layers sequentially rather than in parallel channel-split branches and retains the 1:4 FFN bottleneck, distinct from LightConv (Wu et al., 2019), which replaces self-attention entirely with lightweight convolutions and lacks a global attention branch, and distinct from MobileBERT (AIMech:MobileBERTCompactTaskAgnosticBERTWithBottlenecks), which narrows intra-block channels via linear bottlenecks rather than flattening the FFN and splitting channels across parallel attention and convolution extractors.
Claims & Grounded Evidence
4 Claims
Mechanism
#lite-transformer-flattened-bottleneck-for-1d-attention
For typical 1D NLP sequence lengths N (~20-30 tokens), the standard Transformer 1:4 bottleneck between attention O(4 N d^2 + N^2 d) and FFN O(2 * 4 N d^2) causes the context-agnostic FFN to consume the majority of Mult-Adds (754M vs 582M in Transformer-Base); Lite Transformer flattens this bottleneck so that attention and FFN share the same channel dimension, shifting the computational budget to the context-capturing module.
Mechanism
#lite-transformer-parallel-channel-split-long-short-range-attention
Long-Short Range Attention (LSRA) splits the input along the channel dimension into two equal d/2 groups (reducing branch computation by 2x) and processes them side-by-side through a global multi-head self-attention branch (which specializes in off-diagonal long-range token dependencies) and a local GLU + lightweight depthwise convolution branch (which captures diagonal neighborhood correlations), followed by FFN channel mixing.
Application
#lite-transformer-mobile-translation-summarization-and-compression
Under mobile constraints (500M/100M MACs), Lite Transformer outperforms vanilla Transformer by +1.2/+1.7 BLEU on WMT'14 En-Fr and by +3.1 BLEU on IWSLT'14 De-En (<100M MACs), reduces Transformer-Base computation by 2.5x (and model size by 18.2x to 9.7 MB when combined with pruning and 8-bit k-means quantization), lowers WikiText-103 test perplexity by 1.8, and beats Evolved Transformer by +0.5 BLEU with 19,567x lower CO2 design cost.
Limitation
#lite-transformer-quadratic-global-branch-scaling-and-manual-kernel-limitation
Because the global extractor branch retains standard O(N^2 (d/2)) pairwise dot-product self-attention on half of the channels and the local branch uses manually specified layer-wise convolution kernel sizes (e.g., [3, 5, 7, 31x3]), quadratic sequence-length scaling still re-emerges for very long sequences N >> d.
⚠️ Curated Limitations & Negative Results
- The global self-attention branch retains O(N^2 * d/2) complexity with respect to sequence length N, so quadratic attention cost still dominates when N is very large.
- Layer-wise depthwise convolution kernel schedules (e.g., [3, 5, 7, 31x3] for translation vs [15, 15, 31x4, 63x6] for language modeling) must be manually configured per task family.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
broad_match | Lite Transformer is a neural sequence-to-sequence and language modeling architecture built on the Long-Short Range Attention (LSRA) block. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r817-primary-pdf
|
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, Song Han
|
primary paper
full_text
|
ID: https://arxiv.org/bibtex/2004.11886
sha256:587acac1ca4a…
|
r817-arxiv-abs
|
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, Song Han
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2004.11886
sha256:27f9b9d07d1a…
|
r817-metadata-cr
|
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, Song Han
|
primary paper
abstract_only
|
ID: https://arxiv.org/bibtex/2004.11886
sha256:7989876dc756…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T15:00:00Z
Updated: 2026-10-19T15:00:00Z
Source File:
data/curated/LiteTransformerWithLongShortRangeAttention.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories