LocoProp Layerwise Local Loss Backpropagation Optimization

LocoProp (Local Loss Optimization) is a layerwise neural network optimization framework that enhances each standard backpropagation pass by decomposing an M-layer network into M decoupled, parallel convex layerwise optimization subproblems solved with first-order updates: W_m^{new} = argmin_{W_tilde} { D(W_tilde y_hat_{m-1}, a_m) + R(W_tilde, W_m) }. After a single forward-backward pass fixes layer inputs y_hat_{m-1} and pre-activation gradients grad_{a_hat_m} L(y, y_hat), LocoProp-S sets pre-activation gradient-descent targets a_m = a_hat_m - gamma grad_{a_hat_m} L(y, y_hat) with squared loss 1/2 ||W_tilde y_hat_{m-1} - a_m||^2, while LocoProp-M sets post-activation mirror-descent targets y_m = y_hat_m - gamma grad_{a_hat_m} L(y, y_hat) and minimizes the convex Bregman matching loss D_{F_m}(y_m, f_m(W_tilde y_hat_{m-1})) induced by the convex integral function F_m of the layer's non-decreasing transfer function f_m = grad F_m plus proximal regularizer 1/(2 eta) ||W_tilde - W_m||_F^2. In both variants, T = 1 local iteration recovers exact backpropagation with effective learning rate eta_e = eta * gamma, while T > 1 local iterations approximate implicit/second-order preconditioned updates without forming matrix inverses.