LoRA+ (Asymmetric Learning Rate Low-Rank Adaptation)

LoRA+ is an optimization algorithm for Low-Rank Adaptation (LoRA) of large-width neural networks that sets distinct, asymmetrically scaled learning rates for the two low-rank adapter matrices A in R^{r x n} (initialized with variance Theta(n^{-1})) and B in R^{n x r} (initialized to zero) in W = W^* + (alpha / r) B A. Based on an infinite-width scaling analysis (n -> infinity with fixed rank r << n) showing that efficient feature learning—where both linear feature-update terms delta_t^1 = B_{t-1} Delta Z_A^t and delta_t^2 = Delta B_t Z_A^{t-1} are Theta(1) without exploding activations—requires eta_A = Theta(n^{-1}) and eta_B = Theta(1), LoRA+ enforces a fixed learning-rate ratio eta_B = lambda * eta_A with lambda >> 1 (typically lambda in [2^1, 2^4]) and tunes only the base learning rate eta_A at zero additional computational or memory cost.