Vector-based Random Matrix Adaptation (VeRA)

Vector-based Random Matrix Adaptation (VeRA) is a parameter-efficient fine-tuning (PEFT) method that reparameterizes the low-rank weight update as h = W_0 x + Delta W x = W_0 x + Lambda_b B Lambda_d A x, where B in R^{m x r} and A in R^{r x n} are frozen, Kaiming-initialized random matrices shared across all adapted layers and regenerated on-the-fly from a single random number generator (RNG) seed, while only the layer-specific diagonal scaling vectors b in R^{m} (with Lambda_b = diag(b), initialized to zero) and d in R^{r} (with Lambda_d = diag(d), initialized to a non-zero constant d_{init}) are trainable. Across L_{tuned} adapted layers of dimension d_{model}, VeRA reduces the trainable parameter count from LoRA's 2 L_{tuned} d_{model} r to L_{tuned} (d_{model} + r) while still allowing Delta W = Lambda_b B Lambda_d A to be merged directly into W_0 with zero inference latency.