Home / Curated / Layers & Modules / Projected Attention Layers

Projected Attention Layers

Layers & Modules AIMech:PALsProjectedAttentionLayers DRAFT

Aliases: PALs, BERT and PALs, Projected Attention Layer, Parallel Projected Multi-Head Attention Adapter

A task-specific parallel multi-head attention adaptation layer for multi-task Transformer models that transforms each layer's hidden states via h^{l+1} = LN(h^l + SA(h^l) + TS(h^l)) with TS(h^l) = V^D g^l(V^E h^l), where task-specific encoder V^E in R^{d_s x d_m} and decoder V^D in R^{d_m x d_s} (d_s < d_m) are shared across all Transformer layers for a given task while g^l(.) applies layer-specific low-dimensional multi-head self-attention in R^{d_s} (omitting the final output matrix W^o), trained jointly with annealed task-sampling exponent alpha(e) = 1 - 0.8 (e-1)/(E-1).

Identity Notes & Concept Scope Introduced by Asa Cooper Stickland and Iain Murray (ICML 2019, PMLR 97:5986-5995, arXiv:1902.02671) as Projected Attention Layers (PALs). Distinct from Houlsby bottleneck adapters and AdapterFusion (AIMech:AdapterFusion), which insert serial feed-forward down/up-projection modules within each layer; from AdaptFormer (AIMech:AdaptFormer), which places parallel bottleneck MLP branches alongside FFN sublayers in Vision Transformers; and from HyperFormer (AIMech:HyperFormerSharedHypernetworkAdapter), which generates serial adapter weights via a shared task-conditioned hypernetwork.

Claims & Grounded Evidence

4 Claims

Mechanism #pals-parallel-projected-multihead-attention-with-shared-encoder-decoder

PALs augment each L-layer BERT block in parallel as h^{l+1} = LN(h^l + SA(h^l) + V^D g^l(V^E h^l)), sharing task-specific encoder V^E in R^{d_s x d_m} and decoder V^D in R^{d_m x d_s} across all 12 layers while omitting the multi-head output matrix W^o inside g^l, so that each task requires 2 d_m d_s + 12 x 3 d_s^2 parameters (1.81M parameters per task at d_m=768, d_s=204 vs. d_s=84 when V^E and V^D are not shared across layers).

supports Source: BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning · Locator: Section 3.3 & Table 1, Equations (8), (11), pp. 4-5

Mechanism #pals-annealed-task-sampling-schedule

To balance large and small datasets during multi-task training without late-stage task interference, PALs sample a batch from task i with probability p_i propto N_i^alpha where the exponent alpha(e) = 1 - 0.8 (e - 1)/(E - 1) anneals linearly from 1.0 (dataset-proportional sampling) at epoch 1 to 0.2 at final epoch E.

supports Source: BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning · Locator: Section 4.1, Equations (12)-(14), p. 5

Application #pals-glue-multitask-parameter-efficiency-and-rte-transfer

On eight GLUE benchmark tasks with BERT-base, PALs (d_s=204) require only 1.13x total parameters (~7x fewer parameters than eight separately fine-tuned BERT-base models) while achieving a higher mean GLUE test score (80.4 vs. 79.6) and improving RTE test accuracy from 66.4% to 76.0%.

supports Source: BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning · Locator: Section 5, Tables 2-3, pp. 6-7
supports Source: BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning (Crossref metadata) · Locator: ICML 2019 bibliographic record

Limitation #pals-shared-layer-computation-and-joint-training-limitation

Because PALs insert task-specific branches at every layer of BERT-base and jointly fine-tune the shared backbone across all tasks simultaneously, they require simultaneous access to all tasks at training time and cannot share forward-pass layer activations across tasks on the same input text unless PALs are restricted to the top layers.

supports Source: BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning · Locator: Section 2.2, Section 5.2 & Section 6, pp. 3, 8

⚠️ Curated Limitations & Negative Results

Inserting PALs at every Transformer layer prevents sharing intermediate forward-pass computations across tasks evaluated on the same input sequence unless PALs are restricted to the final half of the network.
Joint multi-task training of the shared BERT backbone with PALs requires access to all tasks at training time and exhibits higher variance across random seeds under annealed sampling.
In extreme low-parameter regimes (e.g., d_s = 60, 1.5M total adaptation parameters), PALs offer no significant advantage over simple low-rank linear layers or a fully shared model.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	PALs are parallel low-dimensional multi-head attention layers added to multi-task Transformer neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r712-primary-pdf`	BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning Asa Cooper Stickland, Iain Murray	primary paper full_text	ID: https://dblp.org/search/publ/api?q=BERT+and+PALs+Projected+Attention+Layers+for+Efficient+Adaptation&h=5&format=xml sha256:3f8e9009fea1…
`r712-arxiv-abs`	BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning (arXiv abstract) Asa Cooper Stickland, Iain Murray	primary paper abstract_only	ID: https://arxiv.org/abs/1902.02671 sha256:34ba131cb13f…
`r712-metadata-cr`	BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning (Crossref metadata) Asa Cooper Stickland, Iain Murray	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=BERT+and+PALs+Projected+Attention+Layers+for+Efficient+Adaptation&h=5&format=xml sha256:35cfbd22c742…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T13:15:00Z

Updated: 2026-10-19T13:15:00Z

Source File: data/curated/PALsProjectedAttentionLayers.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories