Home / Curated / Algorithms / NOLA

NOLA

Algorithms AIMech:NOLA DRAFT

Aliases: NOLA (Compressing LoRA Using Linear Combination of Random Basis), Random-Basis Bilinear Low-Rank Adaptation, Sub-Rank-One Random Basis LoRA Reparameterization

A parameter-efficient fine-tuning and model-compression method that overcomes the rank-one parameter lower bound of Low-Rank Adaptation (LoRA) by reparameterizing the rank-r factor matrices A in R^{m x r} and B in R^{r x n} as linear combinations of frozen pseudo-randomly generated basis matrices A = sum_{i=1}^k alpha_i A_i and B = sum_{j=1}^l beta_j B_j, optimizing and storing only the scalar PRNG seed and the (k + l) mixture coefficients (alpha, beta) to decouple trainable parameter count from both matrix dimensions (m, n) and rank r.

Identity Notes & Concept Scope Distinct from standard LoRA (AIMech:LoRA, whose parameter count r(m + n) is lower-bounded by m + n at rank r = 1 and tied to layer dimensions), from VeRA (AIMech:VeraVectorBasedRandomMatrixAdaptation, which scales a single random matrix pair via diagonal vectors d in R^m and b in R^r whose parameter count still depends on layer dimension m and rank r), and from PRANC (which forms a direct linear combination of full m x n random matrices requiring O(k d^2) compute and spanning only a k-dimensional linear subspace rather than NOLA's O(k d r) compute and bilinear cross-basis subspace). Formulated by Koohpayegani, Navaneet, Nooralinejad, Kolouri, and Pirsiavash (ICLR 2024, arXiv:2310.02556).

Claims & Grounded Evidence

4 Claims

Mechanism #nola-bilinear-random-basis-reparameterization-mechanism

NOLA generates frozen random basis matrices {A_i in R^{m x r}}_{i=1}^k and {B_j in R^{r x n}}_{j=1}^l from a single scalar PRNG seed and parameterizes the weight update as Delta W = (sum_{i=1}^k alpha_i A_i) x (sum_{j=1}^l beta_j B_j), requiring only k + l trainable scalar coefficients independent of m, n, and r and achieving compression ratios strictly above LoRA's rank-one limit m n / (m + n).

supports Source: NOLA: Compressing LoRA using Linear Combination of Random Basis · Locator: Section 2 Proposed Method: NOLA, Eq. 1-2, p. 3

Mechanism #nola-compute-speedup-and-subspace-coverage-over-pranc

Compared to full-matrix random basis combination (PRANC) on a d x d weight matrix with k total coefficients, NOLA reduces per-iteration training complexity from O(k d^2) to O(k d r) (a speedup factor of d / r, or 4.6x wall-clock speedup on ResNet18), while its bilinear A x B factorization spans a much higher-dimensional subspace of R^{d^2} than PRANC's k-dimensional linear subspace.

supports Source: NOLA: Compressing LoRA using Linear Combination of Random Basis · Locator: Section 2 Efficiency of NOLA compared to PRANC, p. 4 & Appendix A.1-A.2 Figure 2 and Tables 8-9, p. 15-16

Application #nola-quantized-coefficients-and-llm-vit-adaptation-application

NOLA supports post-training and STE quantization-aware training of the scalar coefficients (alpha, beta) down to 3-4 bits while keeping the random basis matrices A_i, B_j in FP16, and fine-tunes GPT-2 on E2E/DART/WebNLG, 8-bit LLaMA-2-70B on Alpaca/MMLU with 95% fewer parameters than rank-1 LoRA (0.57M vs 12.94M), and ViT-B/ViT-L on few-shot image classification with 1/3 the parameters of rank-4 LoRA.

supports Source: NOLA: Compressing LoRA using Linear Combination of Random Basis · Locator: Section 2-3.4, Tables 1-7, p. 3-8

Limitation #nola-basis-multiplication-training-overhead-limitation

During training, NOLA incurs extra matrix-multiplication overhead to contract the coefficient vectors (alpha, beta) with the basis tensors {A_i} and {B_j} at each forward and backward pass (unless the random basis is shared across layers), whereas at inference time A x B is merged offline into W with zero latency overhead.

supports Source: NOLA: Compressing LoRA using Linear Combination of Random Basis · Locator: Section 3.1 Training Time and Memory of NOLA & Table 2, p. 5-6

⚠️ Curated Limitations & Negative Results

Contracting basis tensors with scalar coefficients during training adds a small per-batch compute overhead when each layer uses a distinct random basis unless basis matrices are shared across layers.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	NOLA reparameterizes low-rank weight adaptation matrices in deep neural networks using linear combinations of random basis matrices.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r688-primary-pdf`	NOLA: Compressing LoRA using Linear Combination of Random Basis Soroush Abbasi Koohpayegani, K. L. Navaneet, Parsa Nooralinejad, Soheil Kolouri, Hamed Pirsiavash	primary paper full_text	ID: https://dblp.org/search/publ/api?q=2310.02556&format=xml sha256:2210d43a859a…
`r688-arxiv-abs`	NOLA: Compressing LoRA using Linear Combination of Random Basis (arXiv abstract) Soroush Abbasi Koohpayegani, K. L. Navaneet, Parsa Nooralinejad, Soheil Kolouri, Hamed Pirsiavash	primary paper abstract_only	ID: https://arxiv.org/abs/2310.02556 sha256:dc8dd23bb28c…
`r688-metadata-cr`	NOLA: Compressing LoRA using Linear Combination of Random Basis (Crossref metadata) Soroush Abbasi Koohpayegani, K. L. Navaneet, Parsa Nooralinejad, Soheil Kolouri, Hamed Pirsiavash	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=2310.02556&format=xml sha256:057320b7643f…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:51:00Z

Updated: 2026-10-19T12:51:00Z

Source File: data/curated/NOLA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories