LISA (Layerwise Importance Sampled AdamW)
Algorithms AIMech:LISALayerwiseImportanceSampledAdamW DRAFT
Aliases: LISA, Layerwise Importance Sampled AdamW, Layerwise Importance Sampling AdamW, Layerwise Importance Sampling for Memory-Efficient LLM Fine-Tuning
A memory-efficient full-rank optimization algorithm for large language models that emulates the skewed layerwise update-norm distribution of Low-Rank Adaptation without low-rank subspace restrictions by permanently unfreezing the bottom embedding and top language-modeling head layers while periodically sampling a small fixed number gamma of intermediate Transformer layers every K iterations to update with AdamW.
Identity Notes & Concept Scope
Distinct from BAdam (AIMech:BAdamBlockCoordinateAdamFullParameterOptimization), which partitions all model layers into D blocks of equal status and updates one block per K-step epoch via cyclic or random-reshuffling block coordinate descent; LISA is grounded in the empirical skewness of layerwise weight norms w^{(ell)} = (1/T) sum_{t=1}^T ||theta_t^{(ell)}||_2 in LoRA (where embedding and LM head norms exceed intermediate layers by 10^2x), assigning importance-sampling probabilities {1.0, gamma/N_L, ..., gamma/N_L, 1.0} so E and H remain continuously active while gamma intermediate layers are sampled every K steps. Distinct from LoRA (AIMech:LoraLowRankAdaptation), which freezes base weights and constrains updates to rank-r adapter matrices across all layers.
Claims & Grounded Evidence
5 Claims
Mechanism
#lisa-skewed-weight-norm-and-importance-sampling-mechanism
Observing that LoRA's layerwise mean weight norms w^{(ell)} = (1/T) * sum_{t=1}^T ||theta_t^{(ell)}||_2 are skewed by factors of hundreds toward the token-embedding and language-modeling head layers relative to intermediate Transformer blocks, LISA matches expected layerwise step sizes eta_t^{(ell)} = tilde_eta_t^{(ell)} * (tilde_w^{(ell)} / w^{(ell)}) in full-rank space by setting layer activation probabilities {p_ell}_{ell=1}^{N_L} = {1.0, gamma/N_L, ..., gamma/N_L, 1.0} and, every K iterations, freezing all intermediate layers except gamma randomly sampled layers that are updated alongside the embedding and head layers via AdamW.
Mechanism
#lisa-memory-and-iteration-speedup-mechanism
Because LISA introduces no auxiliary adapter modules, deletes redundant intermediate activations during backpropagation, and allocates gradients and AdamW states only for the embedding layer, LM head, and gamma active intermediate layers (E+H+gamma L), it reduces LLaMA-2-7B fine-tuning memory from 59 GB (full training) to 21-23 GB (fitting on a single 24 GB GPU), reduces LLaMA-2-70B memory to 71-75 GB vs 79 GB for LoRA rank 128, and achieves a 2.9x per-iteration speedup over full-parameter training (1.5x over LoRA).
Scope
#lisa-regularized-convex-convergence-scope
Viewing AdamW on loss f(w) as randomized block-coordinate Adam on the scaled quadratic-regularized objective f^{reg}(w) = f(w) + (1/2) * w^T S w with positive semidefinite diagonal matrix S, Theorem 1 guarantees that for convex smooth f on a bounded domain with bounded stochastic gradients, LISA iterates satisfy (1/T) * sum_{t=1}^T f^{reg}(w_t) - f_*^{reg} <= O(1 / sqrt(T)).
Application
#lisa-llm-instruction-tuning-and-continual-pretraining-application
Across instruction tuning of TinyLlama-1.1B, Mistral-7B, LLaMA-2-7B, and LLaMA-2-70B on Alpaca-GPT4, GSM8K, and PubMedQA, and continual pre-training on 1.5B tokens of OpenWebMath, LISA outperforms LoRA by 10%-35% on MT-Bench (reaching 6.72 vs 6.10 for LoRA and 6.25 for full training on LLaMA-2-70B) and matches full-parameter continual pre-training on GSM8K at less than half the GPU memory cost (26 GB vs 59 GB).
Limitation
#lisa-forward-pass-base-model-memory-and-uniform-middle-sampling-limitation
During the forward pass, LISA still requires the full pre-trained base model weights to reside in GPU memory, and its default E+H+gamma L schedule samples all intermediate layers with uniform probability gamma / N_L rather than adapting middle-layer sampling probabilities to task- or architecture-specific importance.
⚠️ Curated Limitations & Negative Results
- Requires holding the full base model parameters in memory during the forward pass unless paired with weight quantization.
- Approximates intermediate-layer importance with uniform sampling probability gamma / N_L across all middle layers.
- Requires tuning the sampling period K and active middle-layer count gamma alongside the learning rate.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | LISA is a layerwise importance-sampled AdamW optimization algorithm for fine-tuning and continual pre-training of deep Transformer language and diffusion models. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r640-primary-pdf
|
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, Tong Zhang
|
primary paper
full_text
|
ID: https://dblp.org/rec/conf/nips/PanLDPZH024
sha256:e79aa3ab643c…
|
r640-arxiv-abs
|
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, Tong Zhang
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2403.17919
sha256:3a249d44fba4…
|
r640-metadata-cr
|
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, Tong Zhang
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/conf/nips/PanLDPZH024
sha256:497b885086b2…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T12:03:00Z
Updated: 2026-10-19T12:03:00Z
Source File:
data/curated/LISALayerwiseImportanceSampledAdamW.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories