HAT (Hardware-Aware Transformers for Efficient Natural Language Processing)
Architectures AIMech:HATHardwareAwareTransformersForEfficientNLP DRAFT
Aliases: HAT, Hardware-Aware Transformers, HAT SuperTransformer with Arbitrary Encoder-Decoder Attention
HAT (Hardware-Aware Transformers) is a hardware-aware Neural Architecture Search (NAS) framework for sequence-to-sequence encoder-decoder Transformers that breaks the information bottleneck between encoder and decoder via Arbitrary Encoder-Decoder Attention (allowing each decoder layer to attend to the concatenated key-value representations of the last 1, 2, or 3 encoder layers with zero extra parameters), enables layer-wise Heterogeneous Transformer Layers (varying embedding dimensions, FFN hidden dimensions, attention head counts, and decoder layer count across ~10^15 candidate SubTransformers), trains a single weight-sharing SuperTransformer by uniformly sampling SubTransformers and slicing front-prefix weight subtensors, and executes evolutionary search guided by a target-device MLP Latency Predictor to extract specialized SubTransformers meeting strict hardware latency constraints.
Identity Notes & Concept Scope
Distinct from Evolved Transformer (So et al., 2019), which searches homogeneous cell structures without hardware latency feedback at >10,000x higher GPU cost, distinct from Once-for-All (Cai et al., 2020), which targets convolutional neural networks via progressive shrinking rather than encoder-decoder Transformers with arbitrary cross-attention spans, and distinct from DynaBERT (AIMech:DynaBERTDynamicBERTWithAdaptiveWidthAndDepth), which adapts encoder-only BERT width and depth via importance-ordered head/neuron rewiring and distillation.
Claims & Grounded Evidence
4 Claims
Mechanism
#hat-arbitrary-encoder-decoder-attention-and-heterogeneous-layers
HAT defines a heterogeneous design space of ~10^15 encoder-decoder SubTransformers in which each layer independently selects its FFN inner dimension and attention head count while Arbitrary Encoder-Decoder Attention lets each decoder layer attend to the last 1, 2, or 3 encoder layers by concatenating their key and value vectors along the sequence dimension without adding trainable parameters.
Mechanism
#hat-weight-sharing-supertransformer-and-latency-predictor-search
HAT trains a single weight-sharing SuperTransformer by uniformly sampling SubTransformers at each step and updating their front-slice weight subtensors, and pairs the SuperTransformer with a 3-layer MLP hardware latency predictor (trained on 2000 measured architecture-latency pairs with ~0.1 ms/0.1 s RMSE) during evolutionary search under a hard hardware latency budget.
Application
#hat-hardware-specialized-machine-translation-speedup
On WMT'14 En-De, WMT'14 En-Fr, WMT'19 En-De, and IWSLT'14 De-En across Raspberry Pi-4 ARM CPU, Intel Xeon CPU, and Nvidia Titan Xp GPU, HAT achieves up to 3x latency speedup and 3.7x smaller model size over Transformer-Big while cutting NAS search cost by 12,041x compared to Evolved Transformer, automatically discovering deep-and-thin models for ARM CPU versus shallow-and-wide models for GPU.
Limitation
#hat-post-search-retraining-and-device-profiling-limitation
Although the SuperTransformer ranks candidate SubTransformers accurately, achieving peak final translation BLEU requires retraining the discovered SubTransformer from scratch for 40K steps, and each new target hardware platform requires profiling ~2000 architectures to fit a device-specific latency predictor.
⚠️ Curated Limitations & Negative Results
- Extracted SubTransformers require post-search retraining from scratch (e.g., 40K steps) to reach full stand-alone BLEU accuracy.
- Deploying to a new hardware target requires collecting ~2000 on-device latency measurements to train the device-specific MLP latency predictor.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
broad_match | HAT is a hardware-aware neural architecture search and weight-sharing SuperTransformer framework for sequence-to-sequence neural networks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r813-primary-pdf
|
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, Song Han
|
primary paper
full_text
|
ID: https://arxiv.org/bibtex/2005.14187
sha256:5162696f5c4d…
|
r813-arxiv-abs
|
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, Song Han
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2005.14187
sha256:e11c0c2afd1f…
|
r813-metadata-cr
|
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, Song Han
|
primary paper
abstract_only
|
ID: https://arxiv.org/bibtex/2005.14187
sha256:43bd7fcaf89d…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T14:56:00Z
Updated: 2026-10-19T14:56:00Z
Source File:
data/curated/HATHardwareAwareTransformersForEfficientNLP.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories