Quantization-Aware Low-Rank Adaptation (QA-LoRA)

Quantization-Aware Low-Rank Adaptation (QA-LoRA) is a joint low-bit quantization and parameter-efficient fine-tuning algorithm for large language models that partitions each column j in {1, ..., D_out} of a pretrained weight matrix W in R^{D_in x D_out} into L contiguous groups of size g = D_in / L, quantizing each group l in {1, ..., L} with its own scaling factor alpha_{l, j} and zero-point beta_{l, j} into N-bit integers W_hat_{i, j} in {0, 1, ..., 2^N - 1} via W_tilde_{i, j} = alpha_{floor(i/g), j} (W_hat_{i, j} - beta_{floor(i/g), j}), while constraining all row vectors of the low-rank adapter matrix A within each group l to be identical. Implemented via parameter-free non-overlapping group summation (or scaled 1D average pooling) A(x)_l = sum_{r=1}^g x_{(l-1)g + r} on the input vector x in R^{D_in} followed by low-rank matrices L_1 in R^{L x D_int} and L_2 in R^{D_int x D_out}, QA-LoRA computes the training forward pass y = x^T W_tilde + s A(x)^T L_1 L_2 and, after fine-tuning, merges the adapter losslessly into the quantized model simply by shifting the group-wise zero-point parameters beta^{new}_{l, j} = beta_{l, j} - s (L_1 L_2)_{l, j} / alpha_{l, j} while keeping W_hat in N-bit integer precision.