Home / Curated / Architectures / MoLE (Mixture of LoRA Experts)

MoLE (Mixture of LoRA Experts)

Architectures AIMech:MoLEMixtureOfLoRAExperts DRAFT

Aliases: Mixture of LoRA Experts, MoLE Hierarchical LoRA Composition, Layer-Wise Gated Mixture of Pre-Trained LoRA Experts

A hierarchical modular composition architecture for pre-trained Low-Rank Adaptation (LoRA) modules that treats each layer of N frozen trained LoRAs as a distinct expert, concatenates and normalizes their layer outputs E_{Delta theta_i}(x) in R^{L x d} into E_Omega(x) in R^{N L d}, projects them via a learnable gating embedding e in R^{(N L d) x N} and temperature-scaled softmax G(eps_i) = exp(eps_i / tau) / sum_j exp(eps_j / tau) to weight each LoRA expert's output, and regularizes gating entropy across M blocks via the balancing loss L_balance = -log(prod_{i=0}^{N-1} q^(i)).

Identity Notes & Concept Scope Distinct from LoraHub (AIMech:LoraHub, which optimizes static scalar weights per LoRA module via gradient-free black-box optimization), from LoRA-Flow (AIMech:LoRAFlow, which computes token-level gates directly from the pre-LoRA hidden state x^l_t without concatenating LoRA expert outputs or optimizing a log-product gating balancing loss), and from LoRAMoE (AIMech:LoRAMoE, which trains LoRA experts from scratch inside FFN blocks with a localized balancing constraint to prevent world knowledge forgetting during SFT rather than composing separately pre-trained frozen LoRAs). Formulated by Wu, Huang, and Wei (ICLR 2024, arXiv:2404.13628).

Claims & Grounded Evidence

4 Claims

Mechanism #mole-output-conditioned-hierarchical-layer-gating

For a set of N frozen trained LoRAs Omega = {Delta theta_i}_{i=0}^{N-1} at a given block or layer, MoLE computes each LoRA's output E_{Delta theta_i}(x) in R^{L x d}, concatenates and normalizes them into E_Omega(x) = Normalization(E_{Delta theta_0}(x) oplus ... oplus E_{Delta theta_{N-1}}(x)) in R^{xi} (xi = N L d), computes logits eps = Flatten(E_Omega(x))^T e in R^N, and combines expert outputs via O(x) = F_theta(x) + sum_{i=0}^{N-1} [exp(eps_i / tau) / sum_j exp(eps_j / tau)] E_{Delta theta_i}(x).

supports Source: Mixture of LoRA Experts · Locator: Section 3.2 Mixture of LoRA Experts, Eq. 7-13, p. 5-6

Mechanism #mole-gating-balancing-loss-and-proportional-masking

To prevent gating entropy collapse toward a single early-stage dominant LoRA expert during training, MoLE adds the gating balancing loss L_balance = -log(prod_{i=0}^{N-1} q^(i)) where q^(i) = (1/M) sum_{k=1}^M exp(eps_i^k / tau) / sum_{j=1}^N exp(eps_j^k / tau) across M gated blocks, and at inference time supports manual masking of undesired LoRAs with proportional redistribution of the remaining gating weights without retraining.

supports Source: Mixture of LoRA Experts · Locator: Section 1 Figure 1, Section 3.3 Eq. 14-16 & Section 5 Table 7 and Figure 8, p. 1, 6, 8-9, 12, 14

Application #mole-multimodal-and-nlp-composition-application

MoLE is applied to multi-subject text-to-image diffusion composition with DreamBooth on Stable Diffusion V2.1 and to multi-task NLP composition on FLAN-T5 across WMT translation, Struct-to-Text, Closed-Book QA, Big-Bench Hard (BBH), and NLI, outperforming normalized linear arithmetic, SVDiff, PEMs, and LoraHub.

supports Source: Mixture of LoRA Experts · Locator: Section 4.1-4.2, Tables 1-3 & Section 5 Tables 6-10, p. 6-9, 12-13

Limitation #mole-very-large-expert-array-and-matrix-granularity-limitation

When the number of composed LoRAs scales to very large arrays (such as 128 LoRAs on BBH), composition performance degrades relative to moderate expert counts (e.g., 48 LoRAs), and applying gating at overly fine parameter-matrix granularity (m-MoLE) underperforms block-wise (b-MoLE) and layer-wise (l-MoLE) gating due to excessive control interfering with intra-layer parameter relationships.

supports Source: Mixture of LoRA Experts · Locator: Section 5 Coarse-to-fine gating analysis, Section 6 Limitations & Tables 6 and 9, p. 8-9, 12-13

⚠️ Curated Limitations & Negative Results

Composition accuracy declines when scaling to very large arrays of 128 simultaneous LoRA experts, and parameter-matrix-level gating (m-MoLE) underperforms intermediate block-level (b-MoLE) and layer-level (l-MoLE) gating granularities.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	broad_match	MoLE is a hierarchical mixture-of-experts neural network composition architecture over pre-trained low-rank adapter layers.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r686-primary-pdf`	Mixture of LoRA Experts Xun Wu, Shaohan Huang, Furu Wei	primary paper full_text	ID: https://dblp.org/search/publ/api?q=2404.13628&format=xml sha256:415b8d661e5e…
`r686-arxiv-abs`	Mixture of LoRA Experts (arXiv abstract) Xun Wu, Shaohan Huang, Furu Wei	primary paper abstract_only	ID: https://arxiv.org/abs/2404.13628 sha256:d2cb6e24cd1a…
`r686-metadata-cr`	Mixture of LoRA Experts (Crossref metadata) Xun Wu, Shaohan Huang, Furu Wei	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=2404.13628&format=xml sha256:16ddc5ec05b1…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:49:00Z

Updated: 2026-10-19T12:49:00Z

Source File: data/curated/MoLEMixtureOfLoRAExperts.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories