LoRAMoE
Architectures AIMech:LoRAMoE DRAFT
Aliases: LoRAMoE Plugin with Localized Balancing Constraint, Localized-Balanced Mixture of LoRA Experts for World Knowledge Retention, MoE-Style LoRA Plugin for Alleviating World Knowledge Forgetting
A Mixture-of-Experts-style parameter-efficient fine-tuning architecture that freezes the pre-trained backbone feed-forward linear weights W_0 to preserve parametric world knowledge, introduces N parallel low-rank experts Delta W_{E_i} = B_i A_i gated by a trainable softmax router G(x) = Softmax(x W_g) as o = W_0 x + (alpha / r) sum_{i=1}^N omega_i B_i A_i x, and enforces a Localized Balancing Constraint L_lbc = sigma^2(I o Q) / mu(I o Q) during supervised fine-tuning to partition experts into world-knowledge-aligning experts and downstream-task experts while balancing utilization within each group.
Identity Notes & Concept Scope
Distinct from HydraLoRA (AIMech:HydraLoRA, which shares a single down-projection matrix A across multiple up-projection heads B_k without type-partitioned localized balancing), from MoLE (AIMech:MoLEMixtureOfLoRAExperts, which freezes pre-trained LoRAs and gates over their concatenated layer outputs), and from standard sparsely-gated MoE layers (which replace the entire FFN with full-rank experts and apply a single-distribution global coefficient-of-variation balancing loss). Formulated by Dou et al. (ACL 2024, arXiv:2312.09979).
Claims & Grounded Evidence
4 Claims
Mechanism
#loramoe-ffn-plugin-forward-routing-mechanism
In each Transformer FFN block, LoRAMoE freezes the pre-trained linear weight matrix W_0 in R^{d_in x d_out} and adds N trainable low-rank experts Delta W_{E_i} = B_i A_i (A_i in R^{d_in x r}, B_i in R^{r x d_out}, r << min(d_in, d_out)) weighted by a router G(x) = Softmax(x W_g) so that the layer output is o = W_0 x + (alpha / r) sum_{i=1}^N omega_i B_i A_i x.
Mechanism
#loramoe-localized-balancing-constraint-mechanism
Given the sample-level expert importance matrix Q_{n,m} = sum_{j=1}^{T_m} G(x_j)_n for expert n and training sample m and the binary-group affinity coefficient matrix I_{n,m} = 1 + delta when Type_e(n) = Type_s(m) and 1 - delta when Type_e(n) != Type_s(m) (delta in [0, 1]), LoRAMoE minimizes L_total = L + beta * L_lbc with L_lbc = sigma^2(I o Q) / mu(I o Q), softly specializing subsets of experts to world-knowledge vs. downstream tasks while balancing utilization within each expert group and requiring no task-type labels at inference time.
Application
#loramoe-large-scale-sft-knowledge-retention-application
LoRAMoE is applied to large-scale multi-task supervised fine-tuning of LLaMA-2-7B on 3 million instruction samples spanning reading comprehension, NLI, translation, summarization, coreference, and closed-book QA (TriviaQA, Natural Questions, HotpotQA), improving world-knowledge benchmark scores by an average of 35.3% (up to 63.9%) over vanilla SFT and 30.9% over single-LoRA PEFT while matching or exceeding SFT on downstream tasks.
Limitation
#loramoe-coarse-binary-task-taxonomy-limitation
During training, the Localized Balancing Constraint requires partitioning training datasets into pre-defined coarse task categories (world-knowledge vs. other downstream tasks) to construct the importance coefficient matrix I_{n,m}, and its extension to finer-grained multi-category expert taxonomies and models larger than 7B parameters remains unverified.
⚠️ Curated Limitations & Negative Results
- Requires pre-categorizing training samples into coarse task types (world-knowledge vs. downstream tasks) during training to compute the localized balancing constraint matrix I_{n,m}.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
broad_match | LoRAMoE is a neural network Mixture-of-Experts plugin architecture with low-rank adapter experts. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r687-primary-pdf
|
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=2312.09979&format=xml
sha256:e5bf545cb7b1…
|
r687-arxiv-abs
|
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2312.09979
sha256:373c85cde835…
|
r687-metadata-cr
|
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=2312.09979&format=xml
sha256:67421fddaeae…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T12:50:00Z
Updated: 2026-10-19T12:50:00Z
Source File:
data/curated/LoRAMoE.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories