NOAH (Neural Prompt Search)

NOAH embeds three complementary prompt modules into each of the 12 blocks of a frozen Vision Transformer supernet: VPT-Deep at the block input x = [CLS, P, E] with m learnable tokens P (Eq. 4), LoRA residuals alongside query and key projections Q = x W_q + s x W^down_q W^up_q and K = x W_k + s x W^down_k W^up_k (Eq. 3), and a bottleneck Adapter h' = phi(h W_down) W_up after the normalized MLP output (Eq. 2), training the supernet via uniform subnet sampling over module depths in {3, 6, 9, 12} and embedding dimensions in {1, 5, 10} or {5, 10, 50, 100} with AutoFormer weight entanglement so prefix tokens and low-rank dimensions are maximally shared across subnets.

After supernet training, NOAH performs evolutionary search (crossover of module trajectories between top-k parent subnets and mutation of active depth in {0, 3, 6, 9, 12} and layer dimensions with probability 0.2) under a parameter budget cap (0.64M parameters on VTAB-1k), discovering complementary architectures where Adapter and LoRA concentrate in deep layers (dimension > 4) while VPT spans all depths, and allowing inherited subnets to be deployed directly without retraining (74.1% vs. 75.5% retrained average on VTAB-1k).

Evaluated on ViT-B/16 across the 19-dataset VTAB-1k benchmark (achieving 75.5% average top-1 accuracy with 0.43M average parameters, outperforming Full tuning 68.9%, VPT 72.0%, Adapter 73.9%, and LoRA 74.5%), 5-dataset few-shot visual recognition (Food101, OxfordFlowers102, StanfordCars, OxfordPets, FGVCAircraft), and domain generalization from 16-shot ImageNet to ImageNetV2 (+6.8%), ImageNet-Sketch (+4.8%), ImageNet-A (+5.0%), and ImageNet-R (+5.2%).

NOAH requires 500 epochs of supernet training plus 5 epochs of evolutionary search (and 100 epochs of subnet retraining for peak performance), increasing development cost over training a single prompt module, and in extreme 1-shot or 2-shot regimes or specialized datasets such as EuroSAT it performs only on par with or slightly behind standalone LoRA.