Home / Curated / Algorithms / IA3 Activation Rescaling

IA3 Activation Rescaling

Algorithms AIMech:IA3ActivationRescaling DRAFT

Aliases: IA3, Infused Adapter by Inhibiting and Amplifying Inner Activations

A transformer adaptation method that learns task-specific vectors to rescale attention key/value channels and intermediate feedforward activations while freezing the pretrained network parameters.

Identity Notes & Concept Scope Compared all 442 pinned AIO and 631 native IDs, names, aliases and scoped synonyms; no normalized identity match. Ignored- and hidden-inclusive fixed-string searches of the recovery, older discovery and original Documents checkouts returned 180, 0, 0 lines. All distinct match contents were inspected. Semantic matching covered 40 AIO and 91 native definitions, with every matched definition read. 7 full annotated AIO node/edge/generated contexts and 3 full native records were inspected. Retain IA3 as the specific transformer activation-vector adaptation method. The typeset (IA)^3 is the same method spelling and is included in the identity search; a redundant normalized alias is omitted. Its frozen base with task-specific key/value/feedforward scales differs from additive LoRA updates, speaker-specific LHUC and condition-generated FiLM scale/shift. T-Few combines IA3 with further objectives and a training recipe and is a distinct algorithm scope. Generic fine-tuning model categories do not identify this parameterization.

Claims & Grounded Evidence

5 Claims

Mechanism #attention-channel-rescaling

Each transformer block has learned vectors for the key and value channels in its self-attention and, where present, encoder-decoder attention. In row-activation notation the attention output is softmax(Q*Diag(l_k)*K_transpose/sqrt(d_k))*V*Diag(l_v). The vectors are specific to a task and broadcast across sequence positions; the method does not generate a different gate from every token.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Section 3.3, main PDF p.6, attention expression and broadcast definition; constructed finite differences

Mechanism #feedforward-placement

A further task vector multiplies the intermediate feedforward activation after its nonlinearity and before the output projection. This diagonal scaling is distinct from an additive low-rank weight update. Moving the scale across a nonlinear function generally changes the computation. The described encoder adds L*(d_k+d_v+d_ff) parameters and the decoder adds L*(2*d_k+2*d_v+d_ff), reflecting its two attention modules.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Section 3.3, main PDF p.6, feedforward expression and parameter counts; constructed nonlinearity example

Scope #initialization-and-frozen-base

The added vectors are initialized to ones, which preserves the unadapted computation when first inserted. They are learned thereafter, without a positivity or zero-to-one constraint specified in this source. The base parameters stay fixed during IA3 fitting but still participate in forward and derivative computations. Pretraining the added vectors, as done in T-Few, is a separate learning stage and does not leave them at their initial values.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Sections 3.3–3.4, main PDF pp. 6–7

Mechanism #task-batching-and-absorption

Different examples can use different task vectors while sharing the base matrices in a mixed-task batch. For a fixed task, scales adjacent to a linear projection can instead be absorbed into compatible matrix rows or columns, including the corresponding bias when present. This eliminates the explicit scale operation for that task; one permanently folded matrix does not simultaneously encode arbitrary different task vectors. It also does not permit moving a scale across an intervening nonlinearity.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Section 3.3, main PDF p.6, mixed-task and single-task discussion; direct folding and bias examples

Limitation #finite-comparison-context

The source compares configured adaptation methods on a T0-3B backbone with prompted few-shot classification, additional objectives and method-specific training choices. The results do not establish superiority for every dataset, parameter budget, generative task or implementation. The T-Few recipe additionally specifies losses and pretraining; its complete behavior is not synonymous with the IA3 parameterization.

supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning · Locator: Sections 3.1–3.5, main PDF pp. 4–7; Sections 4–5, pp. 8–10
supports Source: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — Supplementary Material · Locator: Appendix C, supplemental PDF pp. 1–2/printed 17–18; Tables 4–7, supplemental pp. 3–6/printed 19–22

⚠️ Curated Limitations & Negative Results

This local identity denotes the IA3 adaptation parameterization, not the T-Few composite recipe, a T0 checkpoint, FiLM conditioning or ordinary speaker-specific LHUC.
One initialization is an identity transformation, not a permanent positivity or magnitude constraint. Freezing weights does not eliminate base-network computation or activation memory.
Fixed-task scale absorption requires the appropriate linear adjacency and layout. It is not a universal mixed-task merging or latency guarantee.
All sixteen main-paper and eight supplemental pages were visually inspected. The supplement specifies differing comparator settings and a Banking77 exception to the broader T-Few recipe. No author implementation, model fitting, hardware test or empirical replication was executed.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:KnowledgeTransfer` Knowledge Transfer	related_match	The method reuses a pretrained representation for target-task adaptation through a restricted parameterization. The generic transfer strategy is related context rather than an equivalent algorithm.
`AIO:DeepTransferLearning` Deep Transfer Learning	related_match	The demonstrated method adapts deep transformer representations. The pinned target describes a network category, so this is not an algorithm-to-network subclass assertion.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`liu2022ia3`	Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel	primary paper full_text	ID: doi:10.52202/068431-0142 sha256:9c107aa36f12…
`liu2022ia3supp`	Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — Supplementary Material Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel	primary paper full_text	ID: doi:10.52202/068431-0142 sha256:0da202641e40…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-13T16:27:07Z

Updated: 2026-10-13T16:27:07Z

Source File: data/curated/IA3ActivationRescaling.yaml