Home / Curated / Algorithms / S-LoRA

S-LoRA

Algorithms AIMech:SLoRA DRAFT

Aliases: Scalable LoRA Serving with Unified Paging, S-LoRA Concurrent Multi-Adapter Serving

A scalable multi-adapter inference serving algorithm and memory management architecture that decouples batched base-model computation from on-the-fly low-rank adapter evaluation, jointly manages dynamic KV caches and heterogeneous-rank LoRA weights in a single H-page Unified Paging memory pool, executes padding-free non-contiguous gather matrix multiplications (MBGMM and MBGMV), and aligns multi-GPU adapter tensor parallelism with base-model all-reduce collectives.

Identity Notes & Concept Scope Distinct from AIMech:LowRankAdaptation (LoRA), which merges a single adapter W' = W + A B into the base weights and incurs prohibitive weight duplication or serial swapping latency when serving many concurrent adapters, and from AIMech:PagedAttention, which pages only KV cache tensors for a single static weight set. S-LoRA (Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica, MLSys 2024, arXiv:2311.03285) stores thousands of LoRA adapters in host RAM and dynamically prefetches active adapters into GPU memory. Because a per-layer KV cache tensor has shape (S, H) and a LoRA weight matrix has shape (R, H) sharing the hidden dimension H, S-LoRA allocates both from a unified memory pool partitioned into pages of H elements (where sequence length S uses S pages and rank R uses R pages). Custom Triton/CUDA kernels—Multi-size Batched Gather Matrix-Matrix Multiplication (MBGMM) for prefill and Multi-size Batched Gather Matrix-Vector Multiplication (MBGMV) for decode—compute x A B directly over non-contiguous pages across heterogeneous ranks R without padding, while S-LoRA Tensor Parallelism restricts added multi-GPU communication to 5(N-1)Br/N elements.

Claims & Grounded Evidence

5 Claims

Mechanism #slora-unified-paging-interleaved-kv-and-adapter-pool

Exploiting the shared hidden dimension H between KV cache tensors of shape (S, H) and LoRA weight matrices of shape (R, H), S-LoRA's Unified Paging statically allocates a single GPU memory pool of H-element pages in which a KV cache of sequence length S occupies S pages and a LoRA weight tensor of rank R occupies R interleaved non-contiguous pages, eliminating external fragmentation during dynamic adapter loading.

supports Source: S-LoRA: Serving Thousands of Concurrent LoRA Adapters · Locator: Section 5.1 (Unified Paging) & Figure 3, pp. 4-5

Mechanism #slora-mbgmm-and-mbgmv-non-contiguous-heterogeneous-kernels

Rather than merging adapters into W or padding heterogeneous ranks to use dense batched GEMM, S-LoRA batches the shared base projection x W across all requests and executes x A B on the fly using Multi-size Batched Gather Matrix-Matrix Multiplication (MBGMM) during prefill and Multi-size Batched Gather Matrix-Vector Multiplication (MBGMV) during decode directly over non-contiguous page indices.

supports Source: S-LoRA: Serving Thousands of Concurrent LoRA Adapters · Locator: Section 4.1 (Batching) & Section 5.3 (Custom Kernels for heterogeneous LoRA batching on Non-Contiguous Memory), pp. 3-5, Equations (1)-(2) & Figure 1

Mechanism #slora-tensor-parallelism-communication-fusion

Across N GPUs under Megatron-LM tensor parallelism, S-LoRA partitions A_1, B_1 column-wise (with an all-gather on the r-dimensional intermediate vector) and A_2 row-wise, B_2 column-wise (with an all-reduce on the r-dimensional vector and fusion into the base model's final all-reduce), incurring only 5(N-1)Br/N additional communication versus 2(N-1)Bh/N for the base model.

supports Source: S-LoRA: Serving Thousands of Concurrent LoRA Adapters · Locator: Section 6.1 (Partition Strategy) & Section 6.2 (Communication and Memory Cost Analysis), p. 6, Figure 4

Application #slora-application-serving-thousands-of-concurrent-adapters

Evaluated on LLaMA-7B/13B/30B/70B across synthetic power-law workloads and real LMSYS Chatbot Arena traces, S-LoRA serves up to 2,000 concurrent LoRA adapters on a single A100 (80GB) GPU, improving throughput by up to 4x over vLLM-packed and up to 30x over HuggingFace PEFT.

supports Source: S-LoRA: Serving Thousands of Concurrent LoRA Adapters · Locator: Section 7 (Evaluation), Table 3 & Figures 5-8, pp. 6-9

Limitation #slora-limitation-on-the-fly-overhead-for-single-adapter-and-pcie-bandwidth

When serving only one or two static adapters, S-LoRA's separated on-the-fly x A B computation has slightly lower throughput than merging W' = W + A B once into the base model, and under highly uniform request distributions with rapid adapter churn across batches, PCIe host-to-GPU prefetching bandwidth can still bottleneck adapter loading.

supports Source: S-LoRA: Serving Thousands of Concurrent LoRA Adapters · Locator: Section 5.2 (Prefetching and Overlapping) & Section 7.5 (Ablation Study, Figure 9), pp. 5, 9-10

⚠️ Curated Limitations & Negative Results

When serving only 1 or 2 static adapters, computing x A B on the fly incurs minor kernel launch and arithmetic overhead compared to merging W + A B directly into the base weights.
If incoming requests switch across hundreds of distinct high-rank adapters faster than the lookahead queue prediction window, host-to-GPU PCIe transfer latency may not be fully hidden by computation.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	S-LoRA serves thousands of concurrent low-rank adapted transformer neural networks via unified memory paging and heterogeneous gather-matmul kernels.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r670-primary-pdf`	S-LoRA: Serving Thousands of Concurrent LoRA Adapters Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica	primary paper full_text	ID: https://dblp.org/search/publ/api?q=S-LoRA+Serving+Thousands+Concurrent+LoRA+Adapters&format=xml sha256:c8d93e5a2525…
`r670-arxiv-abs`	S-LoRA: Serving Thousands of Concurrent LoRA Adapters (arXiv abstract) Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica	primary paper abstract_only	ID: https://arxiv.org/abs/2311.03285 sha256:464ec25dcbdc…
`r670-metadata-cr`	S-LoRA: Serving Thousands of Concurrent LoRA Adapters (Crossref metadata) Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=S-LoRA+Serving+Thousands+Concurrent+LoRA+Adapters&format=xml sha256:60bae89b510b…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:33:00Z

Updated: 2026-10-19T12:33:00Z

Source File: data/curated/SLoRA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories