LoRA-Fine-Tuning-Aware Quantization (LoftQ)

LoRA-Fine-Tuning-Aware Quantization (LoftQ) is a joint quantization and low-rank adapter initialization framework for quantized parameter-efficient fine-tuning of large language models that minimizes the Frobenius norm discrepancy min_{Q, A, B} ||W - Q - A B^T||_F between each full-precision pretrained weight matrix W in R^{d_1 x d_2} and the sum of its N-bit quantized backbone Q in R^{d_1 x d_2}_N and rank-r low-rank factors A in R^{d_1 x r}, B in R^{d_2 x r}. Initializing A_0 = 0, B_0 = 0, LoftQ alternates for t = 1, ..., T steps between (1) quantizing the residual Q_t = q_N(W - A_{t-1} B_{t-1}^T) using an N-bit lookup-table quantizer q_N(.) (such as NF4 or uniform quantization) and (2) setting A_t, B_t via the exact rank-r truncated singular value decomposition (SVD) of the quantization error W - Q_t = sum_{i=1}^d sigma_{t,i} u_{t,i} v_{t,i}^T with A_t = [sqrt(sigma_{t,1}) u_{t,1}, ..., sqrt(sigma_{t,r}) u_{t,r}] and B_t = [sqrt(sigma_{t,1}) v_{t,1}, ..., sqrt(sigma_{t,r}) v_{t,r}]. During subsequent fine-tuning, Q_T is frozen in N-bit precision while A and B are updated from (A_T, B_T).