# SGLang — full corpus # LLM Wiki An open-source template for building LLM-powered knowledge bases, following [Andrej Karpathy's "LLM Wiki" pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f). You provide raw sources. The LLM reads them, writes structured wiki pages, cross-links everything, and maintains it over time. You never edit the wiki directly — you curate sources and ask questions. ## How It Works The system has three layers: ``` raw/ Sources you collect (articles, transcripts, notes, PDFs) wiki/ LLM-written & maintained pages (summaries, concepts, entities, syntheses) CLAUDE.md Schema that tells the LLM how to structure everything ``` Three operations drive the workflow: | Operation | Trigger | What happens | |-----------|---------|--------------| | **Ingest** | "ingest raw/my-source.txt" | LLM reads the source, creates a summary page, creates/updates concept and entity pages, adds cross-links, updates the index and log | | **Query** | Ask any question | LLM searches the wiki, synthesizes an answer with citations, optionally creates a synthesis page for novel insights | | **Lint** | "lint" or "health check" | LLM audits all pages for orphans, contradictions, missing links, incomplete sections, and low-confidence claims — fixes what it can, reports the rest | ## Quick Start 1. **Clone this repo** ```bash git clone https://github.com/YOUR_USERNAME/llm-wiki.git my-knowledge-base cd my-knowledge-base ``` 2. **Customize CLAUDE.md** for your domain - Update the Purpose section with your topic - Replace the placeholder tagging taxonomy with your own categories - Adjust confidence level descriptions if needed - Everything else (workflows, page formats, linking rules) works as-is 3. **Drop sources into `raw/`** - Text files, transcripts, articles, notes — any plain text - These are immutable once added; the LLM never modifies them 4. **Tell the LLM to ingest** ``` ingest raw/my-first-source.txt ``` The LLM will create summary pages, concept pages, entity pages, cross-links, and update the index. 5. **Ask questions** ``` What are the key differences between X and Y? ``` The LLM answers from the wiki, citing specific pages. 6. **Run health checks** ``` lint ``` The LLM audits the wiki and fixes issues. ## Directory Structure ``` . ├── CLAUDE.md # Schema — the LLM's instructions ├── raw/ # Your source documents (immutable) └── wiki/ ├── index.md # Master catalog of all pages ├── log.md # Append-only activity log ├── dashboard.md # Dataview dashboard (Obsidian) ├── analytics.md # Charts View analytics (Obsidian) ├── flashcards.md # Spaced repetition cards ├── summaries/ # One page per source document ├── concepts/ # Concept and framework pages ├── entities/ # People, tools, organizations, etc. ├── syntheses/ # Cross-cutting analyses and comparisons ├── journal/ # Research/session journal entries │ └── template.md # Journal entry template └── presentations/ # Marp slide decks ``` ## Enhancements This template includes several extras beyond the core wiki pattern: ### Dataview Dashboard (`wiki/dashboard.md`) Live queries that surface low-confidence pages, recent updates, concepts by tag, and pages with the most sources. Requires the [Dataview](https://github.com/blacksmithgu/obsidian-dataview) Obsidian plugin. ### Charts View Analytics (`wiki/analytics.md`) Visual analytics with pie charts, bar charts, and word clouds. Requires the [Charts View](https://github.com/caronchen/obsidian-chartsview-plugin) Obsidian plugin. ### Mermaid Diagrams Use Mermaid code blocks in any wiki page to create flowcharts, sequence diagrams, or concept maps. Native support in Obsidian and GitHub. ### Marp Slides (`wiki/presentations/`) Create slide decks from markdown using [Marp](https://marp.app/). Drop presentation files in this directory. ### Research Journal (`wiki/journal/`) Track your research sessions, experiments, or applied work with the included template. The LLM can reference journal entries when answering queries. ### Spaced Repetition (`wiki/flashcards.md`) Flashcards in the format used by the [Spaced Repetition](https://github.com/st3v3nmw/obsidian-spaced-repetition) Obsidian plugin. Ask the LLM to generate flashcards from any wiki page. ### MCP Server This repo works with Claude Code's MCP server capabilities. Point an MCP-compatible client at this repo and the LLM can read/write the wiki programmatically. ## Customizing for Your Domain The schema in `CLAUDE.md` is domain-agnostic. To adapt it: 1. **Purpose** — Describe your knowledge domain in one paragraph 2. **Tagging taxonomy** — Replace placeholder categories with your own (e.g., for a cooking KB: `cuisine`, `technique`, `ingredient`, `equipment`) 3. **Confidence levels** — Adjust the descriptions to match your domain's evidence standards 4. **Entity types** — Update the entity page description to match what entities mean in your domain (people, tools, companies, etc.) 5. **Journal template** — Customize `wiki/journal/template.md` for your workflow Everything else — page format, linking conventions, workflows, rules — is universal and works across domains. ## Example Domains This template works for any knowledge-intensive topic: - **Research notes** — papers, experiments, methodologies - **Book analysis** — themes, characters, author techniques - **Competitive analysis** — companies, products, market trends - **Course notes** — lectures, readings, key concepts - **Personal development** — frameworks, habits, book summaries - **Technical documentation** — APIs, architectures, design patterns - **Hobby deep-dives** — any subject you want to master ## License MIT # SGLang Knowledge Base An LLM-maintained knowledge base on **SGLang** (github.com/sgl-project/sglang, sglang.io) — a high-performance serving framework for LLMs and multimodal/diffusion models: RadixAttention prefix caching, a frontend DSL, an OpenAI-compatible server, and a deep advanced-features stack (speculative decoding, structured outputs, quantization, TP/PP/EP/DP parallelism + PD disaggregation, LoRA, hierarchical caching) plus a full diffusion (image/video) serving subsystem. Pinned to v0.5.18. ## Concepts - [[concepts/sglang-overview|SGLang Overview]] - [[concepts/installation|Installation]] - [[concepts/sending-requests|Sending Requests]] - [[concepts/server-apis|Server APIs]] - [[concepts/offline-engine|Offline Engine]] - [[concepts/sampling-parameters|Sampling Parameters]] - [[concepts/architecture-and-radixattention|Architecture and RadixAttention]] - [[concepts/frontend-dsl|Frontend DSL]] - [[concepts/server-arguments|Server Arguments]] - [[concepts/router-and-model-gateway|Router and Model Gateway]] - [[concepts/parallelism-and-disaggregation|Parallelism and Disaggregation]] - [[concepts/hierarchical-caching|Hierarchical Caching]] - [[concepts/attention-backends-and-cuda-graph|Attention Backends and CUDA Graph]] - [[concepts/speculative-decoding|Speculative Decoding]] - [[concepts/quantization|Quantization]] - [[concepts/structured-outputs-and-tool-calling|Structured Outputs and Tool Calling]] - [[concepts/lora-and-model-loading|LoRA and Model Loading]] - [[concepts/observability-and-determinism|Observability and Determinism]] - [[concepts/diffusion-serving|SGLang Diffusion Serving]] - [[concepts/diffusion-optimization|SGLang Diffusion Optimization Stack]] - [[concepts/supported-hardware|Supported Hardware]] - [[concepts/supported-models|Supported Models]] - [[concepts/developer-and-benchmarking|Developer Guide and Benchmarking]] - [[concepts/references-and-faq|References, FAQ, and Troubleshooting]] ## Summaries - [[summaries/release-digest|SGLang Release Digest (v0.5.x line)]] ## XL Edition (Pro) Per-item reference depth beyond this edition, in the gated XL layer (13 pages): the complete **server-arguments reference** (435 flags across 38 sections) and the full **environment-variable catalog** (320 vars); the full **Model Gateway** ops reference and the **benchmarking & profiling** toolchain; exhaustive references for **parallelism/disaggregation**, **HiCache**, **speculative-decoding methods**, and the **quantization method×hardware matrix**; and deep **per-hardware tuning** guides (NVIDIA, AMD ROCm, Intel XPU/CPU, Ascend NPU, TPU & others). Agents without Pro access should treat that flag/tuning-level depth as "covered in the XL edition" rather than out of scope. --- title: "Architecture and RadixAttention" type: concept tags: [architecture, caching, foundational, well-established, operator] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-readme-md.md", "raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md", "raw/github_doc-docs-docs-advanced-features-session-radix-cache-mdx.md", "raw/github_doc-docs-docs-advanced-features-hicache-design-mdx.md", "raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md", "raw/github_doc-docs-docs-advanced-features-rfork-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition SGLang's serving architecture is built around three cooperating pieces: a **zero-overhead CPU scheduler** that continuously batches incoming requests, a **paged, chunked-prefill execution engine**, and **RadixAttention** — a prefix-caching mechanism that stores every request's KV cache in a shared radix tree so that requests with common prefixes reuse computation instead of recomputing it (raw/github_doc-readme-md.md). **Sourcing note:** this KB's `raw/github_doc-docs-docs-sglang-*.md` file set (29 files) — despite the naming — is entirely SGLang **Diffusion** (image/video generation) documentation, not the core LLM-serving architecture. There is no single dedicated "architecture.mdx" in the mirrored docs. This page instead synthesizes the runtime/scheduler/cache picture from the README's feature list, `hyperparameter_tuning.mdx` (which documents scheduler log fields and knobs), `session_radix_cache.mdx` and `hicache_design.mdx` (which describe the radix-tree cache design directly), and the memory/scheduling section of `server_arguments.mdx`. ## How It Works ### RadixAttention: prefix caching as a radix tree RadixAttention organizes the KV cache as a **radix tree**: each node corresponds to the KV cache of a consecutive span of tokens in GPU memory, and a path from root to leaf represents one request's prefix. Requests that share a prefix — system prompts, few-shot examples, multi-turn history — share the same tree nodes and avoid recomputing that prefix's KV cache (raw/github_doc-docs-docs-advanced-features-hicache-design-mdx.md). This is the mechanism the README credits with "up to 5x faster inference" in SGLang's original design (raw/github_doc-readme-md.md). Cache eviction is governed by `--radix-eviction-policy` (`lru` default, plus `lfu`, `slru`, `priority`) and `--radix-cache-backend` (pluggable via `register_radix_cache_backend`) (raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md). ### Session-aware radix caching For long-lived multi-turn workloads, `UnifiedRadixCache` supports **session-aware** caching: a client tags every request in a session with the same `session_id`, and SGLang keeps that session's reusable KV ahead of unrelated KV in the eviction order. Session references are *soft protection, not memory pins* — referenced KV can still be evicted if reclaiming unreferenced KV isn't enough. Enable with `SGLANG_ENABLE_UNIFIED_RADIX_TREE=1` plus `--enable-session-radix-cache`; register cache on each request via `session_id`, and call `POST /close_session` when a session ends (including error/cancellation paths) to drop its references (raw/github_doc-docs-docs-advanced-features-session-radix-cache-mdx.md). Eviction order differs by KV component: | Component | What's protected | Eviction order | |---|---|---| | Full attention | prefix path from registered leaf to root | unreferenced nodes first, then referenced nodes with fewer session references, then policy (e.g. LRU) | | Sliding-window attention (SWA) | reusable tail covering the window + page alignment | two LRU passes: unreferenced first, then referenced if needed | | Mamba | reusable state on the registered leaf | two LRU passes: unreferenced first, then referenced if needed | Component cascade rules apply: evicting a Full node also evicts its SWA/Mamba data; evicting SWA also evicts Mamba; evicting a leaf removes all component data on it (raw/github_doc-docs-docs-advanced-features-session-radix-cache-mdx.md). ### HiCache: extending RadixAttention beyond GPU memory HiCache generalizes the same radix-tree idea across a three-tier hierarchy modeled on CPU cache design: **L1 = GPU memory, L2 = host (CPU) memory, L3 = distributed storage** (Mooncake, 3FS, NIXL, AIBrix KVCache, or a file backend). It builds a `HiRadixTree` on top of the RadixAttention tree, where each node additionally records *where* its KV cache lives (raw/github_doc-docs-docs-advanced-features-hicache-design-mdx.md). Workflow per request: local match against L1/L2 → prefetch missing spans from L3 (strategies: `best_effort`, `wait_complete`, `timeout`) → compute → write back hot data to L2/L3 (policies: `write_through`, `write_through_selective`, `write_back`). Key flags: `--enable-hierarchical-cache`, `--hicache-ratio`/`--hicache-size`, `--hicache-storage-backend`. This topic has enough depth to warrant its own page — see the planned `[[concepts/hierarchical-caching]]`. ### The scheduler and continuous batching SGLang uses a **zero-overhead CPU scheduler** (raw/github_doc-readme-md.md) that continuously admits new requests into the running batch as capacity frees up, rather than waiting for a batch boundary. Its live state is visible in server logs, e.g.: ```text Output Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317 ``` - **`#queue-req`** — requests waiting; a healthy range is 100–2000. Persistent `0` means the client isn't submitting fast enough; very large values increase scheduling overhead (raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md). - **`token usage`** — KV cache pool utilization; `>0.9` is good utilization. If usage stays low while requests queue, the scheduler is too conservative (lower `--schedule-conservativeness`, e.g. to `0.3`); if the pool fills and requests get retracted frequently, raise it (e.g. to `1.3`) (raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md). - **`--schedule-policy`** — `fcfs` (default), `lpm` (longest-prefix-match — reorders requests toward more cache hits at the cost of scheduling overhead), plus `random`, `dfs-weight`, `lof`, `priority`, `routing-key` (raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md). - **Chunked prefill** — `--chunked-prefill-size` bounds the tokens processed per prefill chunk (`-1` disables chunking); smaller values reduce prefill memory pressure at the cost of prefill speed for long prompts (raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md). - **Priority scheduling** — `--enable-priority-scheduling` plus `--priority-scheduling-preemption-threshold`, `--schedule-low-priority-values-first`, `--default-priority-value` let higher-priority requests preempt running ones (raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md). ### Memory layout Total GPU memory usage decomposes as `model weights + KV cache pool + CUDA graph buffers + activations`. `--mem-fraction-static` sets `(model weights + KV cache pool) / GPU memory capacity`; SGLang defaults it heuristically (`~0.88` when GPU memory can't be detected), but tuning it up maximizes KV cache capacity (higher concurrency); reserving 5–8 GB for activations is a rule of thumb that's typically sufficient, not a fixed guarantee — validate it against the `available_gpu_mem` value SGLang logs at startup and your workload's actual OOM behavior (raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md). ### Fast instance boot: R-Fork Separately from steady-state serving, **R-Fork** (Tensor Remote Fork) addresses cold-start latency: it loads model weights into a new SGLang instance via zero-copy GPU-to-GPU transfer from an already-running "seed" instance (NCCL, Mooncake TransferEngine, or ModelExpress backends), cutting weight-loading time from minutes to seconds — relevant when scaling out replicas behind a router (raw/github_doc-docs-docs-advanced-features-rfork-mdx.md). ## Key Parameters | Flag | Default | Effect | |---|---|---| | `--schedule-policy` | `fcfs` | Request ordering; `lpm` favors cache hits | | `--schedule-conservativeness` | `1.0` | Higher = fewer retractions, more conservative admission | | `--chunked-prefill-size` | `None` | Max tokens per prefill chunk; `-1` disables | | `--mem-fraction-static` | auto (~0.88) | Fraction of GPU memory for weights + KV cache pool | | `--max-running-requests` / `--max-total-tokens` | `None` | Hard caps on concurrency / KV pool size | | `--page-size` | `1` | Tokens per KV cache page | | `--radix-eviction-policy` | `lru` | `lru`, `lfu`, `slru`, `priority` | | `--enable-hierarchical-cache` + `--hicache-ratio`/`--hicache-size` | off | Extends RadixAttention to host memory (HiCache L2) | | `--enable-session-radix-cache` | off | Session-aware eviction ordering for multi-turn workloads | Full detail and many more flags live in `[[concepts/server-arguments]]` and the source (`python3 -m sglang.launch_server --help`). ## When To Use This architecture, including RadixAttention, is enabled by default for typical causal-LM serving (RadixAttention can be turned off with `--disable-radix-cache`, and prefill-only/embedding-mode workloads can skip physical KV allocation entirely). The tuning knobs matter most when: workloads have long or repeated shared prefixes (favor `lpm` scheduling, larger `--mem-fraction-static` for more cache capacity); multi-turn agentic sessions need cache stickiness (`--enable-session-radix-cache`); or KV working sets exceed GPU memory (HiCache). ## Risks & Pitfalls - Setting `--mem-fraction-static` too high starves activation/CUDA-graph memory and causes OOM; too low limits concurrency and throughput (raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md). - `--schedule-policy lpm` reduces redundant computation but adds scheduling overhead — it's a tradeoff, not a strict win (raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md). - Session-radix-cache references are soft: closing a session does not immediately free its KV, and referenced KV can still be evicted under memory pressure — don't treat `session_id` as a memory guarantee (raw/github_doc-docs-docs-advanced-features-session-radix-cache-mdx.md). - Behind a multi-replica router, each SGLang instance's radix tree is independent — cache hit rate is not global. See `[[concepts/router-and-model-gateway]]` for the gateway's own routing-tree scaling tradeoffs. - HiCache's L3 prefetch/write-back strategies trade latency for hit rate (`best_effort` vs `wait_complete` vs `timeout`); the wrong choice under production SLOs can inflate tail latency (raw/github_doc-docs-docs-advanced-features-hicache-design-mdx.md). ## Related Concepts - `[[concepts/server-arguments]]` — full flag reference including all memory/scheduling flags cited here - `[[concepts/router-and-model-gateway]]` — how RadixAttention cache-locality is used (and degrades) across multiple replicas - `[[concepts/frontend-dsl]]` — the client-side programming model that issues the `fork`/`gen` calls the scheduler batches - `[[concepts/hierarchical-caching]]` (planned) — deeper HiCache treatment - `[[concepts/speculative-decoding]]` (planned) — another engine-level throughput technique - `[[concepts/parallelism-and-disaggregation]]` (planned) — PD disaggregation mentioned above, TP/PP/EP/DP sizing ## Sources - raw/github_doc-readme-md.md - raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md - raw/github_doc-docs-docs-advanced-features-session-radix-cache-mdx.md - raw/github_doc-docs-docs-advanced-features-hicache-design-mdx.md - raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md - raw/github_doc-docs-docs-advanced-features-rfork-mdx.md --- title: "Attention Backends and CUDA Graph" type: concept tags: [architecture, hardware, advanced, well-established, operator, developer] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-attention-backend-mdx.md" - "raw/github_doc-docs-docs-advanced-features-piecewise-cuda-graph-mdx.md" - "raw/github_doc-docs-docs-advanced-features-breakable-cuda-graph-mdx.md" - "raw/github_doc-docs-docs-advanced-features-cuda-graph-for-multi-modal-encod.md" confidence: medium --- # Attention Backends and CUDA Graph ## Definition SGLang supports many interchangeable **attention backend** kernel implementations (FlashInfer, FlashAttention 3/4, Triton, FlashMLA, TRTLLM MLA/MHA, etc.), auto-selected by hardware and model architecture or set explicitly via `--attention-backend`. Separately, SGLang uses **CUDA graph capture** to eliminate per-kernel launch overhead, in three complementary forms: standard decode-time CUDA graphs (fixed batch size), **Piecewise CUDA Graph (PCG)** for prefill/extend with dynamic token counts, **Breakable CUDA Graph** for inserting eager execution islands into an otherwise-captured graph (debugging/compatibility), and **CUDA Graph for the multimodal (ViT) encoder**. ## How It Works ### Attention backend support matrix Selected via `--attention-backend` (auto-selected if unset). Two families: **MHA** (standard multi-head attention: Llama, Qwen, etc.) and **MLA** (multi-head latent attention: DeepSeek, etc.). Multimodal attention is selected separately via `--mm-attention-backend`. **MHA backends** (native page-size support / FP8 KV / FP4 KV / spec topk=1 / spec topk>1 / sliding window / multimodal): - `flashinfer` — page>1 ✅, FP8 ✅, FP4 ❌, spec both ✅, sliding window ✅, multimodal ❌. Default for Ampere/Ada and other pre-Blackwell non-Hopper CUDA (A100, A40, etc.). - `fa3` (FlashAttention 3) — page>1 ✅, FP8 ✅, FP4 ❌, spec both ✅, sliding window ✅, multimodal ✅. Default on Hopper (H100/H200/H20) with CUDA 12.3+. - `fa4` (FlashAttention 4) — native page size 128, FP8 ❌, FP4 ✅, spec both ✅, sliding window ✅, multimodal ✅. - `triton` — page>1 emulated (not native), FP8 ✅, FP4 ✅, spec both ✅, sliding window ✅, multimodal ✅. Fallback when FlashInfer unavailable. - `torch_native` (SDPA) — no native paging, FP8 ✅, FP4 ✅, no spec, no sliding window, multimodal ✅. - `flex_attention` (PyTorch) — no native paging, no FP8, FP4 ✅, nothing else. - `trtllm_mha` — native page 16/32/64, FP8 ✅, FP4 ✅, spec topk=1 ✅ only, sliding window ✅, no multimodal. Optimized for Blackwell (B200); also via `--decode-attention-backend trtllm_mha` (XQA path) on SM90/SM120 (H20/H200/5090) — works best at page size 64. - `dual_chunk_flash_attn` — page>1 native, no FP8/FP4/spec/sliding/multimodal. For ultra-long-context models like Qwen2.5-14B-Instruct-1M. - `hpc_ops` — native page 64 only, FP8 ✅ (required flag), no FP4/spec/sliding/multimodal. Hopper (SM90) only; from Tencent Hunyuan AI Infra's HPC-Ops; requires installing the `hpc` package from source; head_dim 128; q/kv head group 4 or 8. - `aiter` (ROCm), `wave` (ROCm), `ascend` (NPU), `intel_xpu`, `intel_amx` (CPU) — platform-specific backends with varying feature coverage (see full matrix in source). **MLA backends** (native page sizes / FP8 KV / FP4 KV / chunked prefix cache / spec topk=1 / spec topk>1): - `flashinfer` (MLA) — page 1 only, FP8 ❌, FP4 ✅, chunked prefix ✅, spec topk=1 ✅, topk>1 ❌. Default on Blackwell for MLA models generally. - `flashmla` — page 64 only, FP8 ✅, FP4 ✅, chunked prefix ✅, spec topk=1 ✅. - `cutlass_mla` — page 128 only, FP8 ✅, FP4 ✅, chunked prefix ✅, spec topk=1 ✅. - `trtllm_mla` — page 32/64, FP8 ✅, FP4 ✅, chunked prefix ✅, spec topk=1 ✅. Auto-selected for DeepSeek V3 on Blackwell. - `cutedsl_mla` (CuteDSL MLA, Blackwell) — page 32/64, FP8 ✅, FP4 ❌, chunked prefix ✅, spec topk=1 ✅. Decode-only; prefill falls back to `trtllm_mla` when unset. Also the DCP-aware MLA kernel for Kimi K3 (see [[concepts/parallelism-and-disaggregation]]). - `tokenspeed_mla` (Blackwell SM100/SM12x) — page 32/64, FP8 **required**, FP4 ❌, chunked prefix ✅, spec topk=1 ✅. Requires `--kv-cache-dtype fp8_e4m3`. - `fa3` — no native paging, FP8 ❌, FP4 ❌, chunked prefix ✅, spec topk=1 ✅, topk>1 ⚠️ page_size=1 only. Default on Hopper for MLA models. - `triton`, `fa4`, `ascend` (MLA) — additional platform variants. Page-size constraints are hard for several backends: FlashInfer MLA (1), FlashMLA (64), Cutlass MLA (128), TRTLLM MLA (32 or 64), CuteDSL MLA (32 or 64, decode-only), TokenSpeed MLA (32 or 64, Blackwell + FP8 only), TRTLLM MHA (16/32/64), HPC-Ops (64). Backends without native paging can still emulate `page_size > 1` at a wrapper layer (expanding page tables to per-token indices) — the "native" column distinguishes true in-kernel paging from this emulation. **Page size semantics**: page size = tokens grouped per KV-cache block. Prefix-cache matching requires a *complete* page — a 32-token prompt with `page_size=64` won't be cached at all (pages aren't padded); a 65-token prompt with `page_size=64` caches only the first 64 tokens, discarding the 65th. `page_size=1` gives maximum prefix reuse (token-level matching) at some kernel-performance cost; prefer `page_size > 1` when raw attention throughput matters more than prefix-cache hit rate. **Speculative decoding topk**: `topk=1` = classic EAGLE; `topk>1` explores multiple branches and needs backend support in both draft and verify paths (see [[concepts/speculative-decoding]]). ### GDN (Gated Delta Network) linear attention backends GDN is an O(n) linear-attention mechanism used in hybrid models that interleave GDN layers with standard full-attention layers (e.g. Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM). GDN is **not** selected via `--attention-backend` — it activates automatically when the model architecture requires it. Its own kernel backend is `--linear-attn-backend` (default `triton`), overridable per-phase with `--linear-attn-decode-backend`/`--linear-attn-prefill-backend`. On SM100/SM103 + CUDA 13+, SGLang auto-selects FlashInfer for GDN prefill under a specific condition set (Triton base backend, BF16 recurrent state, 128-dim K/V heads, no dynamic chunking, page-major KV layout disabled, `--chunked-prefill-size` in [1, 8192]). Since GDN models are **hybrid**, the full-attention layers still need a standard `--attention-backend`, with platform-specific constraints: Blackwell SM120 (e.g. RTX PRO 6000) needs `triton`/`flashinfer` for prefill (trtllm_mha decode-only); other Blackwell (SM100 B200/GB200) needs `triton`, `trtllm_mha`, or `fa4`; Ascend NPU needs `ascend` only; AMD ROCm recommends `triton`; other CUDA (Hopper/Ampere) auto-selects fine. ### DSA (DeepSeek Sparse Attention) backend Native sparse attention used by DeepSeek V3.2, activated automatically for that architecture and selected via `--attention-backend dsa` (deprecated alias `nsa`). Internally dispatches sub-backends per phase, overridable with `--dsa-prefill-backend`/`--dsa-decode-backend`: `flashmla_sparse` (default prefill on Hopper/Blackwell BF16), `flashmla_sparse_q8` (native FP8 q8×kv8 sparse prefill on Hopper, requires `--kv-cache-dtype fp8_e4m3`), `flashmla_kv` (default for FP8 on Hopper, both phases), `flashmla_auto` (picks sparse or kv variant by KV dtype, prefill only), `fa3` (default decode on Hopper BF16), `trtllm` (default decode on Blackwell BF16, and default for FP8 on Blackwell both phases), `tilelang` (default on AMD ROCm), `aiter` (AMD-specific, needs the aiter package). This is the same `flashmla_sparse`/`flashmla_kv` naming that HiSparse's decode-backend auto-selection refers to (see [[concepts/hierarchical-caching]]). ### Hybrid attention (experimental) Mix different backends for prefill vs. decode when one excels at each — e.g. FA4 prefill + TRTLLM MLA decode on Blackwell: ```bash python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-FP4 --tp 8 \ --attention-backend trtllm_mla --moe-runner-backend flashinfer_trtllm \ --quantization modelopt_fp4 --prefill-attention-backend fa4 ``` If only one of `--prefill-attention-backend`/`--decode-attention-backend` is set, the other inherits `--attention-backend`; if both are set and differ, SGLang auto-enables the hybrid wrapper. With speculative decoding, `--speculative-attention-mode decode` (recommended) routes draft/verify through the decode backend; `--speculative-attention-mode prefill` (default) routes through the prefill backend. Constraints: any `trtllm_mha` backend limits speculative decoding to `--speculative-eagle-topk 1`; paged MHA backends with `--page-size > 1` and `topk > 1` require `flashinfer` specifically; CUDA graph always captures the decode backend, and captures the prefill backend only when `speculative-attention-mode=prefill`. ### Automatic backend selection (CUDA) MHA models: Hopper → `fa3` (CUDA 12.3+); Blackwell → `trtllm_mha` (unless speculative decoding with topk>1); other architectures → `flashinfer` else `triton`. MLA models: Hopper → `fa3` (CUDA 12.3+); Blackwell → `flashinfer` generally, `trtllm_mla` specifically for DeepSeek V3; other architectures → `triton`. ### Piecewise CUDA Graph (PCG) Standard CUDA graphs capture the whole forward pass as one fixed-shape unit — fine for decode (fixed batch), but prefill/extend token counts vary per iteration. PCG splits the computation graph into pieces (roughly one per layer) at registered "split points" (e.g. MoE dispatch ops), capturing each piece separately for a set of pre-defined token-length buckets; at runtime, input is padded to the nearest captured size and each piece replayed — eliminating launch overhead for prefill/extend while still handling dynamic shapes. **Enabled by default** for supported configurations (the old `--enable-piecewise-cuda-graph` flag is deprecated); disable with `--disable-piecewise-cuda-graph`. Args: `--piecewise-cuda-graph-max-tokens` (default: `chunked_prefill_size` for non-MLA, `2048` for MLA — further capped by `--max-total-tokens` if set; Llama-2 models auto-cap at 4096 as a workaround), `--piecewise-cuda-graph-tokens` (explicit capture-size list, auto-generated otherwise), `--piecewise-cuda-graph-compiler {eager,inductor}` (default `eager`), `--enforce-piecewise-cuda-graph` (skip all auto-disable conditions — testing only). Auto-generated capture schedule (increasing step size with token range): 4–32 (step 4), 48–256 (step 16), 288–512 (step 32), 576–1024 (step 64), 1280–4096 (step 256), 4096+ (step 512). Mechanism: `install_torch_compiled()` wraps `model.forward`; `torch.compile(backend=SGLangBackend)` traces an FX graph; `split_graph()` cuts it at `CompilationConfig.split_ops`, leaving those submodules to run eagerly while surrounding submodules are compiled and wrapped in `CUDAPiecewiseBackend`. At runtime the resulting "stitching graph" dispatches eager split-ops and replays each piece's captured CUDA graph. `PiecewiseCudaGraphRunner` orchestrates compile (JIT warmup + Dynamo trace) → capture (largest-to-smallest, one warmup + one graph-record pass per size) → replay (binary search for smallest captured size ≥ actual token count, zero-pad into static buffers, replay, slice output back). Memory: the torch allocator overhead is minimized via a global shared pool reused across runners/sizes, reverse-order (large→small) capture so smaller graphs reuse larger ones' memory, and weak-referenced output tensors on the last subgraph. The dominant cost is non-torch memory (the CUDA graph objects' recorded launch parameters), which scales with the number of captured sizes — why the max-tokens cap is conservative by default. **Auto-disabled** for: certain model architectures (e.g. `DeepseekV32ForCausalLM`), speculative decoding, DP attention, pipeline parallelism (`pp_size > 1`), non-CUDA hardware (ROCm, Ascend), MoE A2A backend, LoRA, multimodal/VLM models, DLLM, deterministic inference, PD disaggregation, and expert distribution recorder/EPLB. `--enforce-piecewise-cuda-graph` bypasses all these checks for testing only. Being torch.compile-based, PCG bugs are typically tracing failures (untraceable ops, dynamic control flow, graph breaks) — the workaround is `--disable-piecewise-cuda-graph`. New JIT/sgl-kernels are usually incompatible with tracing out of the box; wrap them with `register_custom_op` (from `sglang.srt.utils.custom_op`) to make them opaque nodes torch.compile won't try to trace into. ### Breakable CUDA Graph Solves two problems standard CUDA graphs create: debugging is impossible inside an opaque captured graph, and some ops (dynamic control flow, host-device sync, JIT compilation) simply can't be captured at all — previously the only fix was disabling CUDA graphs entirely. Breakable CUDA Graph splits the captured region into multiple segments with eager execution allowed in between, preserving most of the performance benefit while allowing targeted ops to run outside the graph. **Debug mode**: `--debug-cuda-graph` wraps the entire decode forward pass in a single graph break — every op runs eagerly but still through the full capture/replay code path, for debugging without changing model code (eliminates CUDA graph's performance benefit; debugging only). **Production selective breaks**: mark a function `@eager_on_graph(enable=True)` (from `sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph`) so it runs eagerly between captured segments during capture, normally otherwise; or call `break_graph()` inline to force a segment split with no computation. Enable at the environment level (required for `@eager_on_graph` to take effect) with `SGLANG_USE_BREAKABLE_CUDA_GRAPH=1`. Mechanism: capture splits into segments at each `@eager_on_graph` call (end current segment → run function eagerly, recording it and its tensor args for replay → begin new segment). Replay launches each CUDA graph segment, then runs its recorded eager function (re-invoked against the same static buffer references, so it sees each replay's updated values), then launches the final segment. Output writeback handles plain tensors (in-place `copy_()`), structured outputs/dataclasses (tensor fields copied in-place, non-tensor fields replaced), and dicts of tensors similarly. Streams forked from the capture stream (for overlapped computation) are tracked via a hook on `torch.cuda.Stream.wait_stream`, auto-joined before a break and re-forked after. Compatibility: works on both CUDA and ROCm/HIP; unsupported on NPU/CPU/MPS/XPU (there, `--debug-cuda-graph` auto-disables with a warning). Requires `cuda-python` (`pip install cuda-python`) on NVIDIA for reliable stream-capture-status queries (`torch.cuda.is_current_stream_capturing()` has proven unreliable on CUDA); ROCm/HIP uses the portable `torch.cuda` API instead since `cuda-python` isn't available there. **Not compatible** with `SGLANG_MEMORY_SAVER_CUDA_GRAPH`. Overhead per break: one `cudaGraphLaunch`, one eager Python call, one capture begin/end pair — negligible for a small number of breaks relative to the saved launch overhead elsewhere. ### CUDA Graph for the multimodal (ViT) encoder Vision encoders have many layers with fragmented ops (LN, QKV, attention, MLP, residuals per layer → frequent kernel launches), typically run at small batch sizes (launch overhead dominates latency), and have highly variable input token counts (patch counts vary with image/video resolution) — exactly the shape-instability that defeats standard CUDA graphs. `ViTCudaGraphRunner` captures the "blocks + merger + optional deepstack merger" portion of the ViT into a CUDA graph, keyed by sequence length `S` (`graph_key = S`): first occurrence of a given `S` captures a graph, subsequent occurrences replay it. More distinct `S` values means more graph-private memory pools (VRAM cost scales with capture-size diversity, same tension as PCG). All parameter-like tensors (`block_input`/`block_ws`/`block_output`, `cu_full_len`/`cu_window_len` and kk variants, `sin_cos_ws`) become static buffers whose *contents* — not identity — get updated on replay. Attention backend arguments are frozen inside the graph (`TritonAttn` needs `[cu_seqlens, cu_seqlens_kk, max_len]`; `FA3` needs `[cu_seqlens, max_len]`; `max_len` is a frozen int constant; `cu_seqlens` is cached at `create_graph()` time and not updated on later replays) — for a given `graph_key=S`, the segmentation pattern in `cu_seqlens` (and window seqlens) must match exactly, or attention segments the sequence incorrectly. The rotary buffer (`sin_cos_ws`) reallocates larger as `seq_len` grows, bounded by `max_content_len`. Enable with `SGLANG_VIT_ENABLE_CUDA_GRAPH=1`: ```bash SGLANG_VIT_ENABLE_CUDA_GRAPH=1 python3 -m sglang.launch_server --model Qwen/Qwen3-VL-8B-Instruct ``` Can combine with Piecewise CUDA Graph (for the language-model side; PCG is already on by default where supported, so only the tuning flags are needed — the old `--enable-piecewise-cuda-graph` flag is deprecated): ```bash SGLANG_VIT_ENABLE_CUDA_GRAPH=1 python3 -m sglang.launch_server --model Qwen/Qwen3-VL-8B-Instruct \ --piecewise-cuda-graph-max-tokens 4096 --piecewise-cuda-graph-compiler eager ``` Known supported models: Qwen2.5-VL, Qwen3-VL. ### Adding a new attention backend Learn from `python/sglang/srt/layers/attention/triton_backend.py` and `.../flashattention_backend.py`. Without CUDA graph: implement `forward_extend` (prefill, prefill-with-KV-cache, target verification — called once per layer), `forward_decode` (normal decode + draft decode — once per layer), `init_forward_metadata` (once per forward, shared across layers, runs the "plan" step for optimizations like split_kv). With CUDA graph: `init_cuda_graph_state` (once per lifetime, shared buffers), `init_forward_metadata_capture_cuda_graph` (before capture, like `init_forward_metadata` but writes to pre-defined buffers), `init_forward_metadata_replay_cuda_graph` (before replay — critical path, must be fast). Linear-attention kernels (GDN, KDA) follow a different pattern: implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/`, dispatched by `GDNKernelDispatcher`/`KDAKernelDispatcher` rather than `@register_attention_backend`. ## Key Parameters - `--attention-backend`, `--prefill-attention-backend`, `--decode-attention-backend`, `--mm-attention-backend` — backend selection. - `--linear-attn-backend`, `--linear-attn-decode-backend`, `--linear-attn-prefill-backend` — GDN kernel selection. - `--dsa-prefill-backend`, `--dsa-decode-backend` — DSA sub-backend override. - `--speculative-attention-mode {prefill,decode}` — hybrid attention + speculative decoding interaction. - `--disable-piecewise-cuda-graph`, `--enforce-piecewise-cuda-graph`, `--piecewise-cuda-graph-max-tokens`, `--piecewise-cuda-graph-tokens`, `--piecewise-cuda-graph-compiler` — PCG. - `--debug-cuda-graph`, `SGLANG_USE_BREAKABLE_CUDA_GRAPH` — Breakable CUDA Graph. - `SGLANG_VIT_ENABLE_CUDA_GRAPH` — ViT CUDA graph. - `--page-size`, `--kv-cache-dtype` — cross-cutting params that constrain valid backend choices. ## When To Use - Let auto-selection choose the attention backend unless benchmarking shows a specific backend wins for your model/hardware/workload combination. - Use hybrid attention when one backend is measurably better at prefill and another at decode (documented as experimental). - Leave PCG at its default (on) for prefill/extend-heavy workloads on supported configurations; PCG is still experimental (memory overhead grows with the number of captured sizes), so disable it with `--disable-piecewise-cuda-graph` if hitting a torch.compile tracing bug, an unexpected compatibility auto-disable, or a memory issue worth investigating. - Use `--debug-cuda-graph` or `@eager_on_graph`/`break_graph()` specifically when debugging a CUDA-graph-related correctness issue or integrating an op incompatible with graph capture. - Enable ViT CUDA graph for VLM serving with small batches and frequent vision requests, where kernel-launch overhead dominates encoder latency. ## Risks & Pitfalls - Backend page-size constraints are hard, not soft: FlashInfer MLA requires page_size=1, FlashMLA requires 64, Cutlass MLA requires 128 — mismatching `--page-size` against the chosen backend will not silently degrade, it constrains what's even selectable. - A small prompt that doesn't fill a complete page is **entirely excluded** from prefix caching (pages cannot be padded) — this interacts directly with `--page-size` choice and [[concepts/hierarchical-caching]]. - PCG is auto-disabled by many common features (speculative decoding, DP attention, PP, LoRA, multimodal, PD disaggregation, deterministic inference) — a deployment combining any of these loses PCG's benefit rather than erroring, unless `--enforce-piecewise-cuda-graph` is forced (testing only, not recommended for production since it skips real compatibility checks). - Breakable CUDA Graph is incompatible with `SGLANG_MEMORY_SAVER_CUDA_GRAPH` — combining them is not supported. - `--debug-cuda-graph` eliminates CUDA graph's entire performance benefit — it is a debugging tool, not a production safety net. - ViT CUDA graph VRAM cost scales with the number of distinct sequence lengths (`S`) encountered — highly variable image/video resolutions can materially increase graph-private memory pool usage. - Hybrid attention + speculative decoding has narrow support: any `trtllm_mha` backend caps speculative topk at 1, and paged MHA with `page_size > 1` + `topk > 1` requires `flashinfer` specifically — other combinations are unsupported. ## Related Concepts - [[concepts/hierarchical-caching]] — quantized KV cache support and HiSparse's DSA decode-backend selection (`flashmla_sparse`, `flashmla_kv`, `flashinfer_sparse_mla`) are attention-backend-dependent; page-size choice ties directly into prefix-cache behavior. - [[concepts/speculative-decoding]] — draft/verify attention backend selection (`--speculative-draft-attention-backend`, `--speculative-attention-mode`) and topk constraints intersect heavily with this page. - [[concepts/parallelism-and-disaggregation]] — DCP requires specific MLA kernels (`cutedsl_mla`); PD disaggregation auto-disables PCG. - [[concepts/quantization]] — FP8/FP4 KV-cache support varies per attention backend; GEMM backend selection (`--fp8-gemm-backend`, `--fp4-gemm-backend`) is a parallel but distinct kernel-selection axis. ## Sources - raw/github_doc-docs-docs-advanced-features-attention-backend-mdx.md - raw/github_doc-docs-docs-advanced-features-piecewise-cuda-graph-mdx.md - raw/github_doc-docs-docs-advanced-features-breakable-cuda-graph-mdx.md - raw/github_doc-docs-docs-advanced-features-cuda-graph-for-multi-modal-encod.md --- title: "Developer Guide and Benchmarking" type: concept tags: [developer, api, advanced] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-developer-guide-overview-mdx.md", "raw/github_doc-docs-docs-developer-guide-contribution-guide-mdx.md", "raw/github_doc-docs-docs-developer-guide-development-guide-using-docker-mdx.md", "raw/github_doc-docs-docs-developer-guide-development-jit-kernel-guide-mdx.md", "raw/github_doc-docs-docs-developer-guide-quantization-contribution-guide-md.md", "raw/github_doc-docs-docs-developer-guide-serve-backend-plugins-mdx.md", "raw/github_doc-docs-docs-developer-guide-setup-github-runner-mdx.md", "raw/github_doc-docs-docs-developer-guide-release-process-mdx.md", "raw/github_doc-docs-docs-developer-guide-bench-serving-mdx.md", "raw/github_doc-docs-docs-developer-guide-benchmark-and-profiling-mdx.md", "raw/github_doc-docs-docs-developer-guide-evaluating-new-models-mdx.md", "raw/github_doc-docs-docs-developer-guide-msprobe-debugging-guide-mdx.md", "raw/github_doc-docs-docs-supported-models-support-new-models-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition SGLang's developer guide covers contributing to the codebase (environment setup, tests, PR/CI mechanics, code style), extending it (JIT kernels, quantization methods, out-of-tree serve backends), and measuring it (four purpose-built benchmark tools plus a layered profiling stack). Together these define how changes to the runtime get built, tested, and shown to actually help (raw/github_doc-docs-docs-developer-guide-overview-mdx.md). ## How It Works ### Contribution workflow Fork the repo (new contributors don't have direct push access), build from source per [[concepts/installation]], and run `pre-commit run --all-files` before opening a PR — first failures often need a second run to fully auto-fix. Unit tests live under `test/registered/unit/`, mirror the `python/sglang/srt/` source tree, and must use `CustomTestCase` + `register_*_ci(...)` registration so `test/run_suite.py` can discover them; e2e/server tests follow `test/registered/README.md` (raw/github_doc-docs-docs-developer-guide-contribution-guide-mdx.md). CI is gated by the `run-ci` label; PR authors can always `/rerun-failed-ci` on their own PR, and `/rerun-test`/`/rerun-group` target specific tests but do **not** rebuild a PR-local `sglang-kernel` wheel, so they can't validate AOT-kernel changes. Notable style rules: avoid duplicating >5-line snippets, minimize CPU-GPU sync (`tensor.item()`, `tensor.cpu()`), keep functions under ~100 lines, prefer `msgspec.Struct` over `dataclasses`/`attrs`, never use `pickle.loads()`/`recv_pyobj()` on untrusted data (use msgpack/JSON), and for new hardware-specific components always prefer new files (e.g. `allocator_ascend.py`) over drastically changing existing code, with the common (NVIDIA/existing) path listed first in any if/else chain. ### Kernel and quantization contribution Two kernel paths exist: the in-tree **JIT** path (`python/sglang/kernels/jit`, compiled at runtime, uses `tvm-ffi` for Python↔C++ tensor passing, `TensorMatcher`/`SymbolicSize`/`CHECK_HOST`/`LaunchKernel` helpers) for lightweight kernels without heavyweight C++ dependencies, and the **AOT `sglang-kernel`** path (`python/sglang/kernels/aot/`) for CUTLASS-class kernels needing wheel packaging and Torch operator registration; FlashInfer-based kernels use JIT even though they're often heavyweight. `load_jit(...)` with `cuda_wrappers=[...]` compiles and exposes a C++ function as a Python method; `cache_once` (not `functools.lru_cache`, which is not compatible with `torch.compile`) memoizes the compiled module (raw/github_doc-docs-docs-developer-guide-development-jit-kernel-guide-mdx.md). Quantization contributions follow a fixed three-layer split — **Config** (parses/validates params, selects scheme), **Scheme** (weight creation/loading/wiring for Linear/MoE/embedding), **Backend kernel** (hardware-specific execution under `python/sglang/srt/hardware_backend/{gpu,npu}/quantization/`) — to keep format semantics separate from hardware execution; PRs need at least one representative model launched per touched quantization method with a `/generate` sanity check, plus accuracy/benchmark results when numerics or performance can change (raw/github_doc-docs-docs-developer-guide-quantization-contribution-guide-md.md). ### Serve backend plugins An out-of-tree runtime can hook into the `sglang serve MODEL_PATH --model-type BACKEND_NAME` CLI via a `sglang.serve_backends` entry point returning a `ServeBackend(api_version=1, run=..., detect=...)`. Only the `sglang` package may publish a console script literally named `sglang`; extensions keep their own executable as an alias that calls the same backend. Automatic (`--model-type auto`) routing requires exactly one `MATCH` from registered detectors — multiple matches force an explicit `--model-type`, and a detector must never initialize accelerators or load weights just to answer `detect()` (raw/github_doc-docs-docs-developer-guide-serve-backend-plugins-mdx.md). ### Benchmarking tools Four tools operate at different levels of the stack (raw/github_doc-docs-docs-developer-guide-benchmark-and-profiling-mdx.md): | Tool | HTTP server? | Scheduler? | Use case | |---|---|---|---| | `bench_serving` | yes (async client) | yes (indirect) | realistic online serving (TTFT/TPOT/ITL); **default choice** | | `bench_one_batch_server` | yes | yes | end-to-end single-batch latency incl. HTTP+scheduler overhead | | `bench_offline_throughput` | no | yes (direct `Engine`) | max throughput without HTTP overhead | | `bench_one_batch` | no | no (`ModelRunner` directly) | kernel-level latency of one static batch | `bench_serving` (raw/github_doc-docs-docs-developer-guide-bench-serving-mdx.md) is an async HTTP load generator supporting `sglang`/`sglang-native` (`/generate`), `sglang-oai`/`vllm`/`lmdeploy` (`/v1/completions`), `-chat` variants (`/v1/chat/completions`), `sglang-embedding`/`vllm-embedding` (`/v1/embeddings`), `trt` (TensorRT-LLM), and `truss`. Datasets: `sharegpt` (default), `random`, `random-ids`, `image`, `generated-shared-prefix` (with `--gsp-group-distribution {uniform,zipf}` for prefix-popularity shaping), `mmmu`, `speed-bench` (SPEED-Bench speculative-decoding eval), and `agentic-trace` (multi-turn replay). Use `num-prompts >= 5 * max-concurrency` for steady-state measurement; `--request-rate inf` bursts everything at once, a finite rate uses a Poisson arrival process. `--fake-prefill` (with a decode server launched via `--disaggregation-transfer-backend fake`) lets you stress-test pure decode throughput in a PD setup without a real prefill node. ### Profiling PyTorch Profiler is the default tool: set `SGLANG_TORCH_PROFILER_DIR`, then trigger via `bench_serving --profile`, the HTTP `/start_profile`/`/stop_profile` endpoints (`num_steps`, `start_step`, `activities`, `merge_profiles`, `detailed_annotations`), or `python3 -m sglang.profiler`. `detailed_annotations=true` folds per-request/per-KV-length aggregates (`sq`, `sqsq`, `sqsk`, `sk`, prefixed `c_`/`g_` for context/generation) into the trace's `step[...]` GPU-stream markers for roofline analysis without per-request detail. PD disaggregation requires profiling prefill and decode workers **separately** (`--profile-prefill-url` / `--profile-decode-url` are mutually exclusive). `--enable-profile-cuda-graph` profiles the one-time decode CUDA-graph capture phase itself (not steady-state execution), always emitting per-kernel CPU/CUDA time summary tables and a CUDA memory snapshot to `graph_capture_profile/`; persisting Chrome trace files on top of that is opt-in via `SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE` (one combined trace per rank, takes precedence if both are set) or `SGLANG_GRAPH_BATCH_CAPTURE` (one trace per captured batch size per rank). Nsight Systems (`nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node`) gives register/shared-memory/low-level CUDA API detail; `--enable-layerwise-nvtx-marker` (with `--disable-cuda-graph`, since CUDA-graph-captured kernels don't emit NVTX) adds per-layer NVTX ranges for Nsight's timeline. Traces open in `chrome://tracing` or https://ui.perfetto.dev/ (raw/github_doc-docs-docs-developer-guide-benchmark-and-profiling-mdx.md). ### Model evaluation and accuracy Before open-sourcing a new model, SGLang strongly recommends running the built-in eval scripts and reporting the results with the launch command and hardware (raw/github_doc-docs-docs-developer-guide-evaluating-new-models-mdx.md): ```bash python -m sglang.test.run_eval --eval-name mmlu --port 30000 --num-examples 1000 --max-tokens 8192 python -m sglang.test.run_eval --eval-name gsm8k --port 30000 --num-examples 200 --num-shots 5 python -m sglang.test.run_eval --eval-name gpqa --port 30000 --num-examples 198 --max-tokens 120000 --repeat 8 python -m sglang.test.run_eval --eval-name humaneval --num-examples 10 --port 30000 python benchmark/mmmu/bench_sglang.py --port 30000 --concurrency 64 # VLMs ``` GSM8K is described as "too easy" for modern models and is treated as a sanity check, not a rigorous benchmark (1–5% run-to-run variance from batching/nondeterminism); the same GSM8K eval is reused as SGLang's general accuracy-regression sanity check in [[concepts/developer-and-benchmarking]]'s contribution flow. For debugging accuracy anomalies and numerical errors more generally, **MSProbe** dumps per-module (`L0`) or per-torch-API (`L1`, or both via `mix`) tensor statistics via `--msprobe-dump-config`, comparing a "problem" run against a "benchmark" run with `msprobe graph_visualize` + TensorBoard, which highlights divergent nodes in red for the user to inspect and narrow down the root cause (raw/github_doc-docs-docs-developer-guide-msprobe-debugging-guide-mdx.md). ### Docker-based dev loop and release process The `.devcontainer` folder lets VS Code auto-launch a dev container with local edits synced in; manual `docker run` variants mount the HF cache and/or the SGLang repo (edited-in-place, since `lmsysorg/sglang:dev` images install in editable mode). VS Code's `launch.json` debugger config runs `sglang.launch_server` as a `debugpy` module target (raw/github_doc-docs-docs-developer-guide-development-guide-using-docker-mdx.md). Self-hosted GitHub Actions runners are configured per-GPU-vendor Docker containers (NVIDIA `nvidia/cuda:13.0.3-devel-ubuntu22.04`, AMD `lmsysorg/sglang:v0.5.8-rocm700-mi30x`) with `config.sh`/`run.sh` and env vars like `HF_HOME`, `SGLANG_IS_IN_CI=true` (raw/github_doc-docs-docs-developer-guide-setup-github-runner-mdx.md). PyPI releases bump `python/pyproject.toml` and `python/sglang/__init__.py`, then run `python/upload_pypi.sh` and cut a GitHub release (raw/github_doc-docs-docs-developer-guide-release-process-mdx.md) — see [[summaries/release-digest]] for the actual v0.5.x cadence and feature history. ## Key Parameters - **`SGLANG_TORCH_PROFILER_DIR`**, **`--enable-profile-cuda-graph`**, **`SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE`** / **`SGLANG_GRAPH_BATCH_CAPTURE`** — profiling output location and CUDA-graph-capture-phase tracing. - **`--num-prompts`, `--max-concurrency`, `--request-rate`, `--dataset-name`** — the core `bench_serving` load-shape controls. - **`--fake-prefill`** — isolates decode-only throughput in a PD setup. - **`SGLANG_EXTERNAL_MODEL_PACKAGE`** (raw/github_doc-docs-docs-supported-models-support-new-models-mdx.md) and **`sglang.serve_backends`** entry points — two documented ways to add a new model or an out-of-tree serve backend without forking SGLang (other extension paths, e.g. JIT kernels and quantization schemes, exist for other kinds of extension — see also [[concepts/supported-models]]). - **`--msprobe-dump-config`** — general numerical/accuracy-debugging hook (not platform-restricted). ## When To Use Use `bench_serving` for any realistic capacity-planning or regression question ("what's my TTFT/throughput at concurrency X") — it's the tool the team defaults to. Reach for `bench_one_batch` only when isolating a single kernel's latency, since it bypasses the scheduler and can OOM at batch sizes a real server handles fine (no dynamic prefill chunking). Use the PyTorch Profiler HTTP endpoints for programmatic, load-driven profiling of a running server, and Nsight Systems + layerwise NVTX when you need register/shared-memory-level detail or a call-stack-to-kernel mapping. Follow the quantization-contribution or serve-backend-plugin paths respectively when adding a new quantization method or connecting an entirely separate inference runtime, rather than hand-rolling either integration. ## Risks & Pitfalls - `bench_one_batch_server` never reaches steady state (single batch) — its per-metric numbers are biased; only `overall_throughput` under `--enable-multi-batch` is authoritative. - `functools.lru_cache` on a JIT kernel loader is not compatible with `torch.compile` — use `sglang.kernels.jit.utils.cache_once` instead. - `/rerun-test` and `/rerun-group` do not build a PR-local `sglang-kernel` wheel — they're insufficient to validate an AOT kernel change bundled with its caller in the same PR. - Profiling with CUDA graphs enabled hides the Python-to-kernel call stack; layerwise NVTX markers are also not emitted for CUDA-graph-captured kernel launches — both require `--disable-cuda-graph` to get full detail, at a real cost to decode performance during the profiling run. - GSM8K accuracy checks have real run-to-run variance (1–5%) from batching and inference nondeterminism — don't treat a single run as ground truth (see [[concepts/references-and-faq]] for the deeper nondeterminism discussion and `--enable-deterministic-inference`). ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/supported-models]] - [[concepts/references-and-faq]] - [[concepts/quantization]] - [[concepts/installation]] - [[summaries/release-digest]] ## Sources - raw/github_doc-docs-docs-developer-guide-overview-mdx.md - raw/github_doc-docs-docs-developer-guide-contribution-guide-mdx.md - raw/github_doc-docs-docs-developer-guide-development-guide-using-docker-mdx.md - raw/github_doc-docs-docs-developer-guide-development-jit-kernel-guide-mdx.md - raw/github_doc-docs-docs-developer-guide-quantization-contribution-guide-md.md - raw/github_doc-docs-docs-developer-guide-serve-backend-plugins-mdx.md - raw/github_doc-docs-docs-developer-guide-setup-github-runner-mdx.md - raw/github_doc-docs-docs-developer-guide-release-process-mdx.md - raw/github_doc-docs-docs-developer-guide-bench-serving-mdx.md - raw/github_doc-docs-docs-developer-guide-benchmark-and-profiling-mdx.md - raw/github_doc-docs-docs-developer-guide-evaluating-new-models-mdx.md - raw/github_doc-docs-docs-developer-guide-msprobe-debugging-guide-mdx.md - raw/github_doc-docs-docs-supported-models-support-new-models-mdx.md --- title: "SGLang Diffusion Optimization Stack" type: concept tags: [parallelism, caching, quantization, hardware, developer, advanced, well-established] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-sglang-diffusion-caching-acceleration-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-teacache-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-cache-dit-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-attention-backends-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-fused-kernels-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-parallelism-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-encoder-parallel-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-ring-sp-performance-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-disaggregation-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-dynamic-batching-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-progressive-resolution-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-quantization-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-performance-optimization-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-profiling-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-ci-perf-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-contributing-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition The SGLang Diffusion optimization stack is the set of performance levers below the API surface described in [[concepts/diffusion-serving]]: caching acceleration (Cache-DiT, TeaCache, plus Spectrum — covered in [[concepts/diffusion-serving]]), attention backend selection, fused CUDA/Triton kernels, multi-axis parallelism (CFG/TP/Ulysses/Ring/encoder/data), disaggregated serving, dynamic batching, progressive-resolution generation, checkpoint/activation/KV-cache quantization, and profiling. The docs split these levers into two decision classes: **output-preserving** (residency, parallelism, kernels, scheduling — should not change model behavior) and **quality-tradeoff** (caching, progressive resolution, quantization — can change the denoising path or numerical representation) (raw/github_doc-docs-docs-sglang-diffusion-performance-optimization-mdx.md). ## How It Works ### Practical order and starting point 1. Pick `--performance-mode` and explicit residency/parallelism flags (see [[concepts/diffusion-serving]] for the deployment cookbook). 2. Choose an attention backend. 3. Add sequence parallelism only when the model and video shape benefit from it. 4. Add dynamic batching for concurrent compatible traffic. 5. Profile before stacking several levers, then add caching/progressive-resolution/quantization only after comparing output quality against an acceptance target. ### Caching acceleration Three complementary strategies reduce denoising cost by skipping redundant computation, at different granularities (raw/github_doc-docs-docs-sglang-diffusion-caching-acceleration-mdx.md): | Strategy | Scope | Mechanism | Best for | | --- | --- | --- | --- | | Cache-DiT | Block-level | Dynamically skips individual transformer blocks | Advanced, higher speedup (up to 1.69×) | | TeaCache | Timestep-level | Skips entire denoising steps by L1 similarity | Simple, built-in | | Spectrum | Timestep-level | Forecasts DiT features to skip steps | Experimental — see [[concepts/diffusion-serving]] | **TeaCache** (raw/github_doc-docs-docs-sglang-diffusion-teacache-mdx.md) tracks the relative L1 distance between the current and previous modulated inputs (`rel_l1 = |current - previous|.mean() / |previous|.mean()`), rescales it with per-model polynomial coefficients, and accumulates it; when the accumulator is below `teacache_thresh` the step is skipped and the cached residual reused, otherwise computation is forced and the accumulator resets. Models with CFG cache separation keep independent positive/negative caches; models without it disable TeaCache automatically whenever CFG is enabled. Support status: Wan2.1 and Z-Image fully supported; Wan2.2 accepted but currently a no-op (coefficients not yet calibrated); HunyuanVideo not supported; Flux and Qwen to-be-supported. **Cache-DiT** (raw/github_doc-docs-docs-sglang-diffusion-cache-dit-mdx.md) layers three techniques: DBCache (dynamic block-level caching from residual differences), TaylorSeer (Taylor-expansion calibration), and SCM (step computation masking). It is a **per-request** switch — requests with different Cache-DiT settings never share a batch: ```python client.images.generate( model="Qwen/Qwen-Image", prompt="A beautiful sunset over the mountains", extra_body={ "enable_cache_dit": True, "cache_dit_params": {"residual_diff_threshold": 0.12, "scm_preset": "fast"}, }, ) ``` `enable_cache_dit` is tri-state (`true`/`false`/unset-follows-server-default via `SGLANG_CACHE_DIT_ENABLED`). `cache_dit_params` accepts DBCache knobs (`Fn_compute_blocks` default 1, `Bn_compute_blocks` default 0, `max_warmup_steps` default 4, `residual_diff_threshold` default 0.24, `max_continuous_cached_steps` default 3, `enable_taylorseer`, `taylorseer_order`), SCM knobs (`scm_preset`: `none`/`slow`(~75% compute, ~1.3×)/`medium`(~50%, ~2×)/`fast`(~35%, ~3×)/`ultra`(~25%, ~4×), plus `scm_compute_bins`/`scm_cache_bins`/`scm_policy`), and a nested `secondary` dict for dual-DiT models' second transformer. For the diffusers backend, `--cache-dit-config ` loads DBCache+TaylorSeer+SCM, parallelism (`ulysses_size`, `tp_size`, `ring_size`, `attention_backend`, `ulysses_anything`, `ulysses_float8`, `ulysses_async`, `extra_parallel_modules`), and quantization (`quant_type`, `exclude_layers`) configs — combinable in one YAML (requires `cache-dit>=1.2.0`). SCM needs ≥8 inference steps (auto-disabled below that, e.g. DMD-distilled models); Cache-DiT cannot combine with `--use-fsdp-inference`. ### Attention backends Backends are enumerated by `AttentionBackendEnum` and selected via `--attention-backend`, which is **strict** for the DiT — auxiliary components (encoders, VAEs) use it when compatible and otherwise fall back to a component default or platform-compatible backend (raw/github_doc-docs-docs-sglang-diffusion-attention-backends-mdx.md). Selection priority: (1) `global_force_attn_backend`, (2) `--component-attention-backends` override for the component being constructed, (3) CLI `--attention-backend`, (4) layer/component default, (5) auto (platform, dtype, installed packages). Notable options: `fa`/`fa3`/`fa4` (FlashAttention, normalized to `fa`), `torch_sdpa` (most portable), `sage_attn`/`sage_attn_3` (lossy, quantized), `sliding_tile_attn`/`video_sparse_attn`/`vmoba_attn`/`sla_attn`/`sage_sla_attn`/`sparse_video_gen_2_attn`/`sol_attn` (CUDA-only sparse backends), `laser_attn`/`block_sparse_attn`/`rain_fusion_attn` (NPU-only). `torch_sdpa` works everywhere (CUDA/ROCm/XPU/MUSA/MPS/NPU); MPS always uses it. `--component-attention-backends` pins a specific module (e.g. `text_encoder=torch_sdpa,transformer=fa`) and is strict — an incompatible override raises rather than silently falling back, except that a sparse self-attention backend uses a compatible dense backend for cross-attention. `--attention-backend-config` supplies backend-specific parameters (JSON/YAML file, JSON string, or `key=value` pairs) such as STA's `mask_strategy_file_path`/`sta_mode`/`skip_time_steps`, VSA's `sparsity`, or Sol-Attn's `tau`/`dense_backend`/`dense_steps`. A per-request `attention_backend_override` sampling param lets one server mix exact/approximate attention across requests (values limited to `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`, and it participates in the dynamic-batch signature) — but it is **rejected** when combined with breakable CUDA graphs, `torch.compile`-baked kernels, sparse server-side backends, or (unless ring-capable) ring parallelism. ### Fused kernels Diffusion transformers and VAEs spend a large share of non-GEMM time on short elementwise chains (adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues); SGLang replaces these with fused CUDA/Triton/CuTe-DSL kernels under `sglang/kernels/ops/diffusion` (raw/github_doc-docs-docs-sglang-diffusion-fused-kernels-mdx.md). Two numerical contracts exist: **bit-exact** kernels (mounted unconditionally, verified against the live eager chain on first use, fall back permanently on mismatch — e.g. the fused LayerNorm+modulate kernel replicates PyTorch's Welford update order) and **not-bit-exact, request-gated** kernels (mounted only for `quality="high"` requests, at batch boundaries, all-or-nothing per transformer; the default `quality="lossless"` runs the unmodified reference chain). 34 operators are registered across 38 implementations. Enable the request-gated set via `--quality high` or the `quality` field on `/v1/images/generations` / `/v1/videos`. Do not combine request-gated fusions with `--enable-breakable-cuda-graph` — SGLang rejects the combination for models with eligible DiT quality sites (models whose high-quality path touches only VAE decode remain allowed, since BCG captures only the DiT). Fusion families under `quality="high"` include Linear+tanh-GELU, LayerNorm+modulate, LTX-2 RMSNorm+modulate, gate RMSNorm, HunyuanVideo strided QK RMSNorm, and SANA-Video BF16-input linear attention. Registration is queryable without importing a backend: `from sglang.kernels.registry import registry; [op for op in registry.ops() if op.startswith("diffusion.")]`. ### Parallelism SGLang Diffusion composes several parallelism axes whose degrees multiply to the total GPU count (raw/github_doc-docs-docs-sglang-diffusion-parallelism-mdx.md): ```text num_gpus = cfg_parallel_degree × tp_size × sp_degree sp_degree = ulysses_degree × ring_degree ``` | Axis | Splits | Communication | Flag | | --- | --- | --- | --- | | CFG parallel | guidance branches | one combine per step | `--cfg-parallel-size` | | Tensor parallel (TP) | weights / attention heads | all-reduce per block | `--tp-size` | | Ulysses SP | sequence ↔ heads | two all-to-alls per attention | `--ulysses-degree` | | Ring SP | sequence rows in attention | neighbor-only K/V rotation | `--ring-degree` | | K/V-gather CP | sequence rows in attention | one K/V all-gather | `--kv-gather-degree` | | Data parallel | requests | none between replicas | `--dp-size` | Constraints: `num_attention_heads % tp_size == 0`; `(num_attention_heads / tp_size) % ulysses_degree == 0` (the **TP-local** head count, not the raw count — 56 heads pass at `tp=2,ulysses=4` but fail at `tp=4,ulysses=4`); ring adds no head constraint but needs the sequence divisible by `ulysses × ring`; ring requires an attention backend declaring `supports_ring_rotation()` (`fa` and `sage_attn` do). Ring does not yet support HunyuanVideo's varlen path or the legacy stacked-QKV `UlyssesAttention` (Wan's VSA). Cross-node friendliness, best to worst: data parallel (nothing on the request path) > CFG parallel (one small exchange/step) > K/V-gather CP > ring SP (designed for it; end-to-end cross-node validated only for MiniMax-H3 so far) > Ulysses SP (needs full-bisection bandwidth, keep intra-node) > TP (worst — per-block all-reduce, already ~70% of sharded kernel time on NVLink). The scaling pattern for crossing nodes is **node-local Ulysses × cross-node Ring**, launched with `--nnodes`/`--node-rank`/`--dist-init-addr` and one command per node changing only `--node-rank`; `--encoder-parallel replicate` is currently required for cross-node deployments (the `auto` fold decision is not node-boundary aware). On exactly 2 peer-accessible GPUs, `SGLANG_DIFFUSION_IPC_A2A` (default on) replaces the Ulysses all-to-all with a CUDA-IPC transport — bitwise identical output, just a different transport. **Encoder parallelism** (raw/github_doc-docs-docs-sglang-diffusion-encoder-parallel-mdx.md) uses otherwise-idle GPUs during text/image encoding via `--encoder-parallel {auto,fold,dp,replicate}`. `fold` TP-shards the encoder (gated at hidden ≥4096 — Qwen3 at 2560 and CLIP-L at 768 got *slower*); `dp` splits the prompt batch across encoder copies (needs hidden ≥1024 **and** more than one prompt per encode call — pair with `--batching-max-size > 1`); `replicate` (the right answer for most single-request latency work) keeps the encoder redundant across replica ranks. The two accelerated modes are mutually exclusive per encoder. `auto` encodes these measured rules and is preferred unless you've benchmarked a pin yourself. Only `replicate` at encoder TP degree 1 is bitwise-identical to single-GPU encoding. **Sequence parallelism** (raw/github_doc-docs-docs-sglang-diffusion-ring-sp-performance-mdx.md) exposes `--sp-degree`, `--ulysses-degree`, `--ring-degree`, and `--sp-attention-mode {ulysses,kv_gather}`. `kv_gather` needs non-causal attention and `--ring-degree 1`; its relative per-rank payload vs. Ulysses is `P/2`, so it's competitive at SP2 but moves ~2×/4× more data at SP4/SP8. TP and SP are orthogonal mesh dimensions and multiply GPU count (`tp_size × sp_degree`); pure SP often wins on one NVSwitch node when weights fit everywhere, while TP+SP trades some latency for meaningfully lower peak memory (measured: 4.6–42.9% slower for 22.9–44.7%+ less peak memory across Qwen-Image/Wan2.2/LTX2.3). FSDP+SP shards weights across the *same* SP workers rather than multiplying GPU count. A measured reference (2× RTX 40-series, Wan2.2-TI2V-5B, `sp=2,ulysses=1,ring=2` vs. single-GPU baseline) showed 1.42× end-to-end speedup and 7.33 GB less peak memory. Cross-node ring is model-specific — passing `--ring-degree > 1` for a model with only single-node Ulysses support may raise or silently compute wrong output; check the model's cookbook first. Cross-node runs are not expected to bit-match single-node runs (different floating-point accumulation order), but the same cross-node deployment must reproduce byte-identical output across repeat requests — use that as the determinism check, not a cross-topology comparison. **Data parallelism**: `--dp-size N` runs N full replicas on `num_gpus/N` GPUs each, own ingress each; requests round-robin, realtime sessions stick to their replica, control ops fan out to every replica; replicas exchange nothing on the request path. Monolithic serving only; one replica per node needs per-replica host addressing (ingress currently binds locally). ### Disaggregation `--disagg-role {monolithic,encoder,denoiser,decoder,server}` splits a pipeline into independent Encoder/Denoiser/Decoder services routed by a central DiffusionServer (raw/github_doc-docs-docs-sglang-diffusion-disaggregation-mdx.md), analogous to LLM PD disaggregation (see [[concepts/parallelism-and-disaggregation]]). Each role launches as a separate `sglang serve` process with `--disagg-server-addr` pointing at the head node; the `server` role needs no GPU. Verified on 8×H200 with `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`: Encoder 2.3s → Denoiser 312.8s (50 steps, layerwise offload) → Decoder 7.1s, ~322s total for an 81-frame 1024×1024 video. **mooncake-transfer-engine** (RDMA) is a hard dependency for tensor transfer between roles; the DiffusionServer only routes lightweight control messages (alloc/push/ready) while tensors move directly peer-to-peer. Result endpoints derive deterministically from `--scheduler-port` (+1 encoder, +2 denoiser, +3 decoder). Multi-machine deployments add `--disagg-p2p-hostname` and `--disagg-ib-device`; multiple instances per role register via semicolon-separated `--*-urls` with `--disagg-dispatch-policy {round_robin,max_free_slots}`. Per-role parallelism flags (`--encoder-tp`, `--denoiser-tp`/`-sp`/`-ulysses`/`-ring`, `--decoder-sp`) auto-derive from `--num-gpus` when unset. ### Dynamic batching An opt-in serving mode (disabled by default, `--batching-max-size 1`) that merges compatible queued native requests into one pipeline batch — separate from LLM continuous batching (raw/github_doc-docs-docs-sglang-diffusion-dynamic-batching-mdx.md): ```bash sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30010 \ --batching-mode dynamic --batching-max-size 8 --batching-delay-ms 5 --enable-batching-metrics ``` `--batching-config` loads a JSON rules file to cap batch size for specific model/resolution combinations. Coverage is per-model (see the compatibility grid in [[concepts/diffusion-serving]]'s compatibility-matrix discussion) — e.g. FLUX.1-dev and Qwen Image support T2I batching; Qwen Image Edit does not. GLM-Image batches only its external AR `/generate` call; DiT denoising and VAE decode stay per-request. `--enable-batching-metrics` surfaces realized batch size, wait time, and merge/reject rates. ### Progressive resolution generation An experimental technique for selected pipelines: run early denoising steps at half (or quarter) spatial resolution, then spectrally upsample before full-resolution steps, cutting the O(n²) attention cost for those steps to ~6% (raw/github_doc-docs-docs-sglang-diffusion-progressive-resolution-mdx.md, based on arXiv 2605.18736). `--progressive-mode {fullres,dct_rewind,dct}` (default `fullres`/disabled; `dct_rewind` recommended), `--progressive-levels` (number of resolution halvings, default 1), `--progressive-delta` (noise-dominated tolerance; higher = more coarse steps = more speedup, default 0.01). Docs-reported speedups (denoising-loop-only timing on the docs' RTX A6000 48GB benchmark setup, not a general expectation): up to 1.63× (FLUX.1), 1.93× (FLUX.2), 2.33× (Z-Image), 2.78× (Wan 2.1 T2V), 1.69× (Qwen-Image), 1.56× (Ideogram 4). `--dit-cpu-offload false` is recommended alongside it — CPU offload's fixed per-step PCIe transfer cost dilutes the speedup. Incompatible with sequence parallelism (raises `RuntimeError`) and `torch.compile` (fixed sequence length breaks the resolution transition); Cache-DiT interaction is experimental. ### Quantization Component path selection and quantized checkpoint materialization are separate capabilities: any component can point at an independent repo, but it is quantized only when the selected loader supports that serialized format (raw/github_doc-docs-docs-sglang-diffusion-quantization-mdx.md). Three resolution paths: an SGLang quantization adapter for the transformer, delegation to Transformers/Diffusers for standard components, or fail-closed for native plain-state loaders that can't restore the format. Notable `quant_family` values: `fp8`/`mxfp4` online quantization (`--quantization {fp8,mxfp4}`; MXFP4 needs ROCm MI350+/gfx95x), `kitchen_int8` online (`comfy-kitchen`, data-free INT8 ConvRot, Turing+), pre-quantized `fp8` via `--transformer-path`/`--transformer-weights-path`, `modelopt-fp8`/`modelopt-nvfp4` (13 published `lmsys/*` checkpoints across FLUX.1/2, Wan2.2, HunyuanVideo, Qwen-Image family), `auto-round` W4A16 (reuses the SRT GPTQ/Marlin backend), `gguf` (community-quantized transformer, CUDA-only, no FSDP/LoRA, TP must align GGML blocks — e.g. MiniMax-H3's transformer shrinks from 61.7 GiB BF16 to 17.5 GiB Q4_K_M), several `comfy-*` families (`comfy-fp8`, `comfy-int8-convrot`, `comfy-w4a8-convrot`, `comfy-w4a4-convrot`, `quanto-int8`), `mxfp8` (NVIDIA/ROCm reuse SRT's dense kernel; Ascend has its own path), `nunchaku-svdq` (SVDQuant, filename-driven auto-detection of precision/rank from `svdq-{int4|fp4}_r{rank}` patterns), and `msmodelslim` (Ascend-only, via `wan_repack.py` conversion to Diffusers format). `--quantization-ignored-layers` keeps matching transformer layer names unquantized during online quantization. **Causal KV-cache quantization** (Quant-VideoGen / QVG) targets long-running autoregressive video sessions where the causal attention cache rivals model-weight size — it compresses cache, not weights. The current chunk and the newest `--kv-cache-quant-keep-recent` completed chunks stay BF16; older stable chunks are packed once with Progressive Residual Quantization (PRQ: iterative k-means centroid + residual, default 1 stage / 128 centroids / int4-or-int2 block-quantized residual). It is lossy and scoped to the LingBot World realtime sliding-window-and-sink path only (not LongLive2 pinned/global sinks or dynamically growing caches). In the initial LingBot measurements (24-frame window, configuration-specific — not a general QVG ratio): int4 used ~47% of dense BF16 KV-cache memory at +~18% per-chunk latency; int2 used ~37% memory with more error. Tuning options: `--kv-cache-quant {off,int4,int2}`, `--kv-cache-quant-stages`, `-centroids`, `-block-size`, `-iters`, `-asymmetric`, `-keep-recent`, `-sink`, `-sink-keep`. ### Profiling PyTorch Profiler (`--profile`, `--num-profiled-timesteps` default 5 after 1 warmup step, `--profile-all-stages` for the full pipeline vs. just denoising) writes trace files viewable at ui.perfetto.dev or chrome://tracing (raw/github_doc-docs-docs-sglang-diffusion-profiling-mdx.md). `--perf-dump-path perf.json` writes a lightweight JSON with stage-level and per-step (`denoise_steps_ms`) timing, useful for spotting which stage dominates or which step has an abnormal spike without a full trace. Nsight Systems (`nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node ... sglang generate ...`) gives kernel-level CUDA detail; `--delay`/`--duration` target specific stages and reduce file size. If a captured trace shows no CUDA kernels, increase inference steps to extend execution time past the capture window. ### CI performance baselines `python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, and writes aggregated stage/denoise-step/E2E timings back into `perf_baselines.json`'s `scenarios` section (raw/github_doc-docs-docs-sglang-diffusion-ci-perf-mdx.md) — usable per-case (`--case qwen_image_t2i`), by regex (`--match`), or for every existing key (`--all-from-baseline`). ### Developer note: contributing to SGLang Diffusion The contribution guide (raw/github_doc-docs-docs-sglang-diffusion-contributing-mdx.md) applies the same quality bar to AI-assisted ("vibe-coded") PRs as any other: no over-commenting, no over-catching, and test end-to-end before submitting. Commit messages follow `[diffusion] : ` (imperative subject, e.g. `[diffusion] cli: add --perf-dump-path argument`). PRs affecting latency, throughput, or memory should include a performance comparison: run the same benchmark before/after with `--perf-dump-path baseline.json` / `new.json`, then `python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json` and paste the resulting Markdown table into the PR. Consider adding a CI testcase (`testcase_configs.py`) for PRs that support a new model, support/fix important features, or significantly improve performance, and update/add the relevant `perf_baselines.json` entry using the CI performance baseline script above if applicable. See [[concepts/diffusion-serving]] for the model-support triage flow this feeds into. ## Key Parameters - `--attention-backend`, `--component-attention-backends`, `--attention-backend-config`, per-request `attention_backend_override`. - `--quality {lossless,high}` — gates the request-gated fused-kernel set. - `--cfg-parallel-size`, `--tp-size`, `--sp-degree`, `--ulysses-degree`, `--ring-degree`, `--kv-gather-degree`, `--sp-attention-mode {ulysses,kv_gather}`, `--dp-size`, `--encoder-parallel {auto,fold,dp,replicate}`. - `--disagg-role {monolithic,encoder,denoiser,decoder,server}`, `--disagg-server-addr`, `--disagg-p2p-hostname`, `--disagg-ib-device`, `--encoder-urls`/`--denoiser-urls`/`--decoder-urls`. - `--batching-mode dynamic`, `--batching-max-size`, `--batching-delay-ms`, `--batching-config`, `--enable-batching-metrics`. - `--progressive-mode {fullres,dct_rewind,dct}`, `--progressive-levels`, `--progressive-delta`. - `--quantization`, `--quantization-ignored-layers`, `--transformer-path`, `--transformer-weights-path`, `--kv-cache-quant {off,int4,int2}` and its `-stages`/`-centroids`/`-block-size`/`-iters`/`-asymmetric`/`-keep-recent`/`-sink*` tuning flags. - `enable_cache_dit` / `cache_dit_params` (per-request), `SGLANG_CACHE_DIT_ENABLED` and the `SGLANG_CACHE_DIT_*` family (server-wide defaults). - `--enable-breakable-cuda-graph`, `--warmup-resolutions` (interacts with request-gated fusions and per-request attention override). - `--profile`, `--num-profiled-timesteps`, `--profile-all-stages`, `--perf-dump-path`. ## When To Use Start with output-preserving levers (performance mode, attention backend, encoder/sequence parallelism, dynamic batching) before touching anything lossy. Reach for Cache-DiT or TeaCache when denoising latency dominates and some quality drift is acceptable — Cache-DiT for the larger, per-request-tunable speedup; TeaCache when a simpler built-in switch suffices and the model is on its supported list. Use disaggregation when encoder/denoiser/decoder have different scaling needs (e.g. many concurrent cheap encodes feeding fewer expensive denoisers) or when RDMA hardware is available to justify the mooncake-transfer-engine dependency. Use progressive resolution for supported single-branch image/video models where the O(n²) attention cost at full resolution dominates and sequence parallelism/`torch.compile` are not already in use. Use quantization (fp8/nvfp4/gguf/etc.) to fit larger models in less memory or bandwidth, matching the specific `quant_family` to the model and hardware in the compatibility table rather than assuming a generic `--quantization` flag covers every component. Profile first when the bottleneck stage is unclear, before stacking multiple levers at once. ## Risks & Pitfalls - Cache-DiT cannot combine with `--use-fsdp-inference`; TeaCache and Spectrum are mutually exclusive; TeaCache on Wan2.2 is accepted but currently a silent no-op (uncalibrated coefficients) — verify actual speedup, don't assume the flag working means acceleration happened. - Request-gated fused kernels (`quality="high"`) plus `--enable-breakable-cuda-graph` is rejected for models with eligible DiT quality sites — BCG warmup would capture the lossless branches and replay would bypass the requested fusions. - `attention_backend_override` is incompatible with breakable CUDA graphs, `torch.compile`-baked kernels, sparse server-side backends, and (unless ring-capable) ring parallelism; `sage_attn`/`sage_attn_3` are lossy and should be quality-validated per workload. - Ulysses head divisibility is computed on the **TP-local** head count, not the raw head count — a configuration can look valid by `H % U == 0` and still fail once TP is factored in. - Ring parallelism does not support HunyuanVideo's varlen path or the legacy stacked-QKV `UlyssesAttention` (Wan's VSA); cross-node ring support is model-specific and passing `--ring-degree > 1` for an unsupported model may raise or silently compute wrong output. - A mis-mapped rank layout (Ulysses not on intra-node NVLink ranks) stays numerically correct but silently loses performance — worth checking first when a sharded run is unexpectedly slow. - K/V-gather CP is restricted to non-causal attention with `--ring-degree 1`; it does not support the legacy varlen `UlyssesAttention` adapter or video sparse attention. - Dynamic batching has no startup probing, runtime learning, OOM retry, or automatic singleton fallback — a merged batch that fails or can't be split fails every request in it, and batch-shape changes mean singleton and dynamic outputs are not bit-exact. - Progressive resolution raises a `RuntimeError` if sequence parallelism is enabled, and is incompatible with `torch.compile` (fixed compiled sequence length can't handle the resolution transition). - Many quantization families are hardware- or feature-restricted in ways that are easy to miss: GGUF is CUDA-only with no FSDP/LoRA and requires GGML-block-aligned TP shards; several `comfy-*` families support offload but not FSDP; MXFP4 requires ROCm MI350+/gfx95x specifically. - Causal KV-cache quantization (QVG) is lossy and currently supports only the LingBot World realtime sliding-window-and-sink path — it does not extend to LongLive2 pinned/global sinks or dynamically growing caches, and fixed-seed frames will not be pixel-identical to BF16 once enabled. - Disaggregated serving has a hard dependency on `mooncake-transfer-engine` for RDMA transfer — it is not an optional convenience for that deployment mode. ## Related Concepts - [[concepts/diffusion-serving]] — the API, deployment cookbook, model support, and Spectrum caching technique this optimization stack underlies. - [[concepts/attention-backends-and-cuda-graph]] — SGLang's LLM-side attention backend and CUDA graph mechanics, contrasted with the diffusion-specific backend list and per-request override here. - [[concepts/parallelism-and-disaggregation]] — SGLang's LLM-side TP/PP/EP/DP and PD disaggregation, contrasted with the diffusion-specific CFG/TP/SP/encoder parallelism and encoder/denoiser/decoder disaggregation here. - [[concepts/quantization]] — SGLang's LLM-side quantization formats, contrasted with the diffusion checkpoint/KV-cache quantization families above. - [[concepts/developer-and-benchmarking]] — SGLang's general profiling and benchmarking tooling, alongside the diffusion-specific profiler flags and CI perf-baseline script here. - [[concepts/sglang-overview]] — the broader SGLang project these optimizations extend. ## Sources - raw/github_doc-docs-docs-sglang-diffusion-caching-acceleration-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-teacache-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-cache-dit-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-attention-backends-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-fused-kernels-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-parallelism-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-encoder-parallel-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-ring-sp-performance-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-disaggregation-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-dynamic-batching-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-progressive-resolution-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-quantization-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-performance-optimization-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-profiling-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-ci-perf-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-contributing-mdx.md --- title: "SGLang Diffusion Serving" type: concept tags: [models, api, install, hardware, user, foundational, well-established] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-sglang-diffusion-index-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-installation-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-api-cli-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-api-openai-api-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-api-post-processing-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-models-with-ar-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-models-with-pe-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-realtime-models-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-spectrum-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-compatibility-matrix-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-deployment-cookbook-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-support-new-models-mdx.md", "raw/github_doc-docs-docs-sglang-diffusion-environment-variables-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition SGLang Diffusion (the `sglang.multimodal_gen` module, branded "SGLang-Diffusion") is a high-performance inference framework for image and video generation, shipped alongside the LLM-serving core described in [[concepts/sglang-overview]]. It provides native SGLang pipelines, diffusers-backend support for models without a native implementation, an OpenAI-compatible HTTP server, and an optimized kernel stack built on precompiled `sgl-kernel` operators plus JIT kernels (raw/github_doc-docs-docs-sglang-diffusion-index-mdx.md). It supports broad model families — Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more — across NVIDIA, AMD, Intel XPU, Ascend, Apple Silicon, and Moore Threads platforms. ## How It Works ### Installation The standard install already bundles the optimized kernel stack (raw/github_doc-docs-docs-sglang-diffusion-installation-mdx.md): ```bash pip install --upgrade pip pip install uv uv pip install "sglang[diffusion]" --prerelease=allow ``` From source: `pip install -e "python[diffusion]"` (or the `uv` equivalent) after cloning the repo. Docker images are published at `lmsysorg/sglang` (tag `:dev` for the standard path, ROCm-specific tags such as `:v0.5.5.post2-rocm700-mi30x` for AMD Instinct GPUs). Platform-specific paths exist for ROCm (AMD), MUSA (Moore Threads — requires swapping in `python/pyproject_other.toml` and installing the `all_musa` extra), Intel XPU, Ascend NPU, and Apple MPS (`brew install ffmpeg uv`, then `uv pip install -e "python[all_mps]"`; diffusion always runs on PyTorch MPS, and the `all_mps` extra's SRT MLX backend dependencies are unrelated — `SGLANG_USE_MLX` has no effect on diffusion). ### CLI: `generate` and `serve` Two entry points cover one-off jobs and persistent serving (raw/github_doc-docs-docs-sglang-diffusion-api-cli-mdx.md): ```bash sglang generate --model-path Qwen/Qwen-Image --prompt "A beautiful sunset over the mountains" --save-output sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --num-gpus 4 --ulysses-degree 2 --ring-degree 2 --port 30010 ``` `sglang generate` runs one generation job and exits; HTTP-server-only arguments are ignored. `sglang serve` starts the HTTP server and keeps the model resident. Both accept `--config config.yaml` (or JSON) for structured configuration, with CLI flags overriding file values. Non-diffusers checkpoints can resolve through a self-hosted "overlay repo" registry (`SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY`, a dev/debug override) that materializes a local componentized copy under `~/.cache/sgl_diffusion/materialized_models/` on first load. Component overrides use `---path` / `--component-paths.` (replace a component's config+weights) or `---weights-path` / `--component-weights-paths.` (replace only weights, keep the base config) — the component key must match `model_index.json` or a native pipeline's registered module name. `--component-residency COMPONENT=MODE` assigns `resident`, `component-offload`, or `layerwise-offload` per component (group selectors `dit`, `text_encoder`, `image_encoder`, `vae`, and `all` are also available; an exact key overrides a matching group, and a group overrides `all`). Legacy per-component flags (`--dit-cpu-offload`, `--layerwise-offload-components`, etc.) remain supported and compose with the new selectors. Use `--backend diffusers` to force vanilla diffusers pipelines when no native implementation exists or a model needs a custom pipeline class (`--trust-remote-code` for custom classes). Health endpoints separate liveness from readiness: `GET /liveness` returns `200` as soon as the HTTP server accepts requests (even during warmup); `GET /health` returns `503` during server-based warmup and `200` once ready; `GET /health_generate` is a compatibility alias that does not itself issue a generation request. Cloud storage for generated outputs (S3-compatible, including MinIO) is configured via `SGLANG_CLOUD_STORAGE_TYPE=s3` plus bucket/endpoint/credential env vars — see Environment Variables below. ### OpenAI-compatible API The HTTP server implements OpenAI-compatible image and video endpoints under `/v1` (raw/github_doc-docs-docs-sglang-diffusion-api-openai-api-mdx.md): - **Images**: `POST /v1/images/generations` (text-to-image; `quality` selects a model-owned sampling level — `lossless` or `high` — see [[concepts/diffusion-optimization]] — but this only takes effect when the target model advertises such a level; omitting `quality`, or sending OpenAI's `auto`, keeps the runtime default instead), `POST /v1/images/edits` (multipart, image + prompt), `GET /v1/images/{image_id}/content` (when `response_format=url` and no cloud storage is configured, a relative URL is returned instead of an absolute one). - **Videos**: `POST /v1/videos` (text-to-video, or image-to-video via multipart `input_reference` or a `reference_url` JSON field), `GET /v1/videos` (list/poll status), `GET /v1/videos/{video_id}/content`. - **Discovery**: `GET /v1/models` and `GET /v1/models/{name}` return the public model name plus diffusion runtime info (num_gpus, task_type, precision, pipeline class); `GET /server_info` also reports `served_model_name`. The resolved public name follows `--served-model-name`, then `--model-id`, then `--model-path` — `--model-id` selects a registered model configuration for checkpoints whose local path can't be identified, it is not a free-form deployment alias. - **LoRA management**: `POST /v1/set_lora` (load/activate one or more adapters by nickname, path, target transformer, strength, and `merge_mode`), `POST /v1/merge_lora_weights`, `POST /v1/unmerge_lora_weights` (must precede switching to a different LoRA), `GET /v1/list_loras`. Regular weights statically merge by default; FSDP-sharded weights use dynamic LoRA to avoid full-gather memory peaks. - **Output quality**: `output-quality` (`maximum`/`high`/`medium`/`low`/`default`) actually defaults to the string `"default"`, which auto-resolves to a compression value of `50` for video and `75` for image; `output-compression` (0–100, takes precedence when set) directly overrides the compression level. PNG ignores both. Note: the cited source itself is inconsistent about this field's own spelling — it's called `output_quality` (underscore) in the request-`quality` discussion elsewhere on the same page but `output-quality`/`output-compression` (hyphenated) in the parameter reference — verify the actual spelling against the API schema/code before relying on it. ### Post-processing Optional steps run after generation and can be combined — frame interpolation runs first (raising frame count), then upscaling runs on every frame (raising resolution) (raw/github_doc-docs-docs-sglang-diffusion-api-post-processing-mdx.md). Frame interpolation (`--enable-frame-interpolation`) uses **only** RIFE 4.22.lite (`elfgum/RIFE-4.22.lite`, auto-downloaded); output frame count follows `(N-1) × 2^exp + 1` where `exp` is `--frame-interpolation-exp` (default `1`). Upscaling (`--enable-upscaling`) uses Real-ESRGAN with architecture auto-detected from checkpoint keys (RRDBNet for quality, SRVGGNetCompact — the default `RealESRGAN_x4.pth` — for speed); `--upscaling-scale` (default `4`) beyond the native 4× applies a bicubic resize after the network output. ### Models with autoregressive or prompt-enhancement stages GLM-Image, Qwen Image Layered, and LongCat-Image ship a bundled autoregressive (AR) stage (raw/github_doc-docs-docs-sglang-diffusion-models-with-ar-mdx.md). Qwen Image Layered and LongCat-Image run the native Qwen2.5-VL component in-process; GLM-Image can instead delegate AR inference to a separately launched SGLang server via `--srt-encoder-url` (with `--srt-encoder-timeout` / `--srt-encoder-connection-timeout` for long-running or flaky links) — the diffusion server sends one HTTP request per AR step, so co-locating both servers on a fast local network is recommended, and startup fails fast if the AR host is offline. ERNIE-Image similarly supports built-in prompt enhancement (native Ministral3 implementation) or an external `--pe-server-url` SGLang PE server; `--layerwise-offload-components pe` streams the in-process PE decoder's layers for memory-constrained deployments (does not apply when `--pe-server-url` is set). Ascend NPU deployments running both servers on the same NPU group need distinct `HCCL_IF_BASE_PORT` / `HCCL_HOST_SOCKET_PORT_RANGE` / `HCCL_NPU_SOCKET_PORT_RANGE` ranges per process. ### Realtime and causal video models Two execution modes generate video incrementally and reuse state across chunks, unlike offline pipelines that denoise one bounded sequence and release state at completion (raw/github_doc-docs-docs-sglang-diffusion-realtime-models-mdx.md): **realtime sessions** (state persists until disconnect, served over the `/v1/realtime_video/generate` WebSocket — e.g. LingBot World, SANA-WM realtime) and **request-based causal generation** (state reused across chunks within one request, then released — e.g. LongLive 2.0, batch-streaming SANA-WM). A causal DiT is not automatically a realtime session model; the pipeline must also register a realtime adapter and implement the WebSocket lifecycle. Requests can override `realtime_causal_sink_size` and `realtime_causal_kv_cache_num_frames`. For LingBot World, `--kv-cache-quant {off,int4,int2}` compresses completed causal KV-cache chunks (lossy, disabled by default) — see [[concepts/diffusion-optimization]] for the underlying Quant-VideoGen mechanism. ### Spectrum acceleration Spectrum forecasts DiT features to skip selected denoising steps (raw/github_doc-docs-docs-sglang-diffusion-spectrum-mdx.md). It is scoped narrowly: available only on native FLUX.1, Wan, HunyuanVideo, and SD3 implementation paths (not `--backend diffusers`, not FLUX.2 yet), reachable only through `sglang generate` and Python sampling parameters (not `sglang serve` or the OpenAI server yet), and mutually exclusive with `--enable-teacache`. ```bash sglang generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A paper boat floating through a misty mountain lake" \ --enable-spectrum --save-output ``` Advanced controls (`--spectrum-window-size` default `2.0`, `--spectrum-flex-window` `0.75`, `--spectrum-warmup-steps` `5`, `--spectrum-m` `4`, `--spectrum-lam` `0.1`, `--spectrum-tau-num-steps` `50`, `--history-size` `100`, `--taylor-order` `1`, `--w` `1.0`) trade speed against fidelity; providing any override implicitly enables Spectrum, but explicit `--enable-spectrum` is preferred for new commands. `--debug` adds shadow-prediction validation work and is not representative of normal latency. ### Supported models and the compatibility matrix Pass a Hugging Face model ID (or local directory) to `--model-path` for `sglang generate`/`sglang serve`, or the equivalent Python loading helper (raw/github_doc-docs-docs-sglang-diffusion-compatibility-matrix-mdx.md — verified against the `main`-branch docs fetched 2026-08-24, not a release-tag-pinned snapshot). Broad families include, on the image side, FLUX (1/2, including klein variants and NVFP4), Z-Image, Qwen-Image (incl. Edit/Layered), LongCat-Image, SD3/3.5, SANA, FireRed-Image, JoyAI-Image, and GLM-Image/Hunyuan3D-2/ERNIE-Image/ideogram-4 as long-tail entries; on the video side, FastWan, SANA-Video, LingBot Video MoE, Wan2.1/2.2 (and TurboWan variants), LongLive 2.0, HunyuanVideo/FastHunyuan, MOVA, MiniMax-H3, Helios, LTX-2/2.5, and Cosmos3; and for realtime/world models, LingBotWorld and SANA-WM. See [[concepts/supported-models]] for SGLang's non-diffusion model coverage. The detailed video optimization matrix cross-references model against TeaCache/Sliding-Tile/Sage/VSA/SLA/SageSLA/SVG2/Laser/BSA/Rain-Fusion support (✅ full, ❌ none, ⭕ not applicable) — see [[concepts/diffusion-optimization]] for what each abbreviation does. A missing checkpoint alias in the matrix does not imply the family is unsupported: the runtime registry also accepts detector-based aliases and local directories matching the same family. ### Deployment and performance-mode presets `--performance-mode` (`manual`/`auto`/`speed`/`memory`, alias `--mode`; default `auto`) applies safe residency/parallelism presets without overriding explicit flags (raw/github_doc-docs-docs-sglang-diffusion-deployment-cookbook-mdx.md). `auto` checks the least-available GPU memory across selected devices: for image workloads with ≥45 GiB per GPU it keeps the DiT resident and layerwise-offloads large auxiliary encoders, below that it keeps the DiT offloaded; it may also enable FSDP+CFG parallelism on validated multi-GPU deployments, and CFG parallelism when the model defaults to CFG and no explicit parallelism policy is set. `speed` favors GPU-resident execution and disables CPU offload by default (may OOM); `manual` keeps every performance flag under explicit control. The quick decision rule: resident + no FSDP for the fastest single-GPU run that fits; component-then-layerwise offload for lower single-GPU memory; FSDP + CFG parallelism + resident sharded components for faster multi-GPU Qwen/Wan CFG generation; SP/Ulysses/Ring for sequence-length scaling; explicit TP for compatibility rather than as a default latency lever. `--served-model-name` decouples the public API identity from the checkpoint mount path (useful across replicas or hosts) — see the OpenAI API section above for resolution order. Startup and readiness probes should point at `/health` with a large failure budget (model loading and compilation can legitimately take minutes); liveness probes should use `/liveness` instead. Benchmark takeaways from the docs: Z-Image and Qwen-Image were faster single-GPU/no-FSDP than FSDP/SP in tested settings; Wan benefited from FSDP replacing DiT offload on validated multi-GPU workloads; component offload mainly helped memory, not latency. Always re-benchmark on the target resolution, frame count, step count, and GPU. See [[concepts/diffusion-optimization]] for the parallelism, caching, and kernel mechanics these modes are built on. ### Supporting new models Adding a model is a triage flow, not a fixed template (raw/github_doc-docs-docs-sglang-diffusion-support-new-models-mdx.md). Read the request path in dependency order: `registry.py` → `configs/pipeline_configs/{model}.py` → `runtime/pipelines/{model}.py` → `runtime/pipelines_core/stages/` → `runtime/models/` (only when the architecture can't be reused). Decide the smallest applicable change: a new checkpoint of an existing family needs only a registry entry and maybe a `SamplingParams`/`PipelineConfig` variant; a new native architecture needs a native pipeline and missing components; a long-tail model can start on the diffusers backend for compatibility first. Prefer native stages directly, then subclassing the narrowest native stage, then a custom single-purpose stage, and only as a last resort an aggregated `BeforeDenoisingStage` (hides multiple responsibilities and bypasses shared offload/profiling/disaggregation/batching hooks). Out-of-tree models can register without touching SGLang's source: call `ModelRegistry.register_model` and `register_pipeline` in an installed package's `__init__.py`, then set `SGLANG_EXTERNAL_MODEL_PACKAGE=` before launching (each process imports it once). Complete native support (not just single-GPU parity) also requires encoder/DiT TP+SP, `ParallelTiledVAE`-based parallel decode, and `LayerwiseOffloadableModuleMixin` layer declarations — see [[concepts/diffusion-optimization]] for what these integrate with. ### Environment variables Runtime configuration is largely environment-driven (raw/github_doc-docs-docs-sglang-diffusion-environment-variables-mdx.md): `SGLANG_DIFFUSION_TARGET_DEVICE` (default `cuda`; also `rocm`/`xpu`/`npu`/`musa`/`mps`/`cpu`), `SGLANG_DIFFUSION_ATTENTION_BACKEND` and `SGLANG_DIFFUSION_ATTENTION_CONFIG` (two separate env-level attention overrides — the former overrides the backend selection, the latter points at a JSON/YAML backend-config file — see [[concepts/diffusion-optimization]]), `SGLANG_DIFFUSION_STAGE_LOGGING`, `SGLANG_DIFFUSION_TORCH_PROFILER_DIR`, `SGLANG_DIFFUSION_CACHE_ROOT` (default `~/.cache/sgl_diffusion`), `SGLANG_DIFFUSION_CONFIG_ROOT`, `SGLANG_DIFFUSION_WORKER_MULTIPROC_METHOD` (`fork`/`spawn`), `SGLANG_DIFFUSION_IPC_A2A` (CUDA-IPC all-to-all for eligible 2-GPU Ulysses groups, default on; `0` forces NCCL) with `_TIMEOUT_MS` (default `10000`) and `_MAX_BUFFERS` (default `16`), and `SGLANG_USE_RUNAI_MODEL_STREAMER`. Platform-specific variables cover Apple MPS (`SGLANG_USE_MLX`, SRT-only, no diffusion effect), ROCm (`SGLANG_USE_ROCM_VAE`, `SGLANG_USE_ROCM_CUDNN_BENCHMARK`), and quantization (`SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND`, `SGLANG_DIFFUSION_ENABLE_W8A8_FP8_GEMM`). Cloud storage uses `SGLANG_CLOUD_STORAGE_TYPE=s3` plus `SGLANG_S3_BUCKET_NAME`/`_ENDPOINT_URL`/`_REGION_NAME`/`_ACCESS_KEY_ID`/`_SECRET_ACCESS_KEY`. CUDA crash debugging exposes `SGLANG_KERNEL_API_LOGLEVEL` (0/1/3/5/10), `_LOGDEST`, `_DUMP_DIR`, `_DUMP_INCLUDE`/`_EXCLUDE`. The `SGLANG_CACHE_DIT_*` family (enable flag, `Fn`/`Bn`/warmup/RDT/max-continuous, TaylorSeer, SCM preset/policy/bins, plus `_SECONDARY_*` variants for dual-transformer models) configures Cache-DiT server-wide defaults — see [[concepts/diffusion-optimization]] for the caching mechanics they control. ## Key Parameters - `--model-path` / `--served-model-name` / `--model-id` / `--model-variant` — checkpoint location, public API name, registry hint, and semantic weight-partition selector, respectively. - `--component-paths.` / `---path`, `--component-weights-paths.` / `---weights-path` — replace a component's config+weights or weights-only. - `--component-residency COMPONENT=MODE` (`resident`/`component-offload`/`layerwise-offload`), plus legacy `--dit-cpu-offload`, `--layerwise-offload-components`, `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, `--dit-layerwise-residency-policy`. - `--performance-mode` / `--mode` (`manual`/`auto`/`speed`/`memory`). - `--num-gpus`, `--tp-size`, `--sp-degree`, `--ulysses-degree`, `--ring-degree`, `--dp-size` (see [[concepts/diffusion-optimization]] for the parallelism math). - `--quality {lossless,high}` — request-level exactness vs. validated-accelerated tradeoff (fused kernels, described in [[concepts/diffusion-optimization]]). - `--prompt`, `--negative-prompt`, `--image-path`, `--num-inference-steps`, `--seed`, `--num-outputs-per-prompt`, `--height`/`--width`/`--num-frames`/`--fps`, `--output-path`/`--output-file-name`/`--save-output`. - `--enable-frame-interpolation` / `--frame-interpolation-exp` / `--frame-interpolation-scale` / `--frame-interpolation-model-path`; `--enable-upscaling` / `--upscaling-scale` / `--upscaling-model-path`. - `--srt-encoder-url` / `--srt-encoder-timeout` / `--srt-encoder-connection-timeout` (AR delegation); `--pe-server-url` (prompt enhancement delegation). - `--enable-spectrum` and the `--spectrum-*` tuning flags. - `--log-requests`, `--log-requests-level {0-3}`, `--log-requests-format {text,json}`, `--log-requests-target`. - `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY`, `SGLANG_EXTERNAL_MODEL_PACKAGE`, `SGLANG_CLOUD_STORAGE_TYPE` + `SGLANG_S3_*`. ## When To Use Use SGLang Diffusion when you need to self-host image/video generation behind an OpenAI-compatible Images API and a documented subset of the OpenAI Videos API (not a blanket drop-in guarantee for video clients), with the same operational shape as SGLang's LLM serving (see [[concepts/sglang-overview]]). Reach for `sglang generate` for one-off/batch jobs and scripted benchmarking, and `sglang serve` for a persistent multi-request server. Use the AR/PE delegation paths (`--srt-encoder-url`, `--pe-server-url`) when GLM-Image's or ERNIE-Image's auxiliary stage needs independent scaling or resources. Use realtime sessions for interactive world models needing live control (camera actions, prompt updates) and request-based causal generation for long videos that stream chunk-by-chunk without needing a persistent WebSocket. Use Spectrum only after validating quality/latency on the exact model, shape, hardware, and sampling settings you plan to deploy — it is explicitly approximate. ## Risks & Pitfalls - The `quality` field in a **video response body** is unrelated to the `quality` sampling parameter — it is fixed Sora-compatible metadata always reported as `"standard"`, not a reflection of what actually ran. - `/health` is not a liveness probe: a long server-based warmup can legitimately hold it at `503` for minutes; use `/liveness` for that purpose instead. - `--model-id` is not a free-form deployment alias — it selects a registry configuration; use `--served-model-name` to control the name clients see. - The AR delegation path (GLM-Image `--srt-encoder-url`) issues one HTTP request per AR step; cross-region encoder/diffusion placement measurably degrades latency, and the diffusion server refuses to start if the AR host is unreachable at startup. - Ascend NPU deployments running two servers (AR + diffusion) on one NPU group must assign non-overlapping `HCCL_*` port ranges or the servers will conflict. - Spectrum is approximate, scoped to only FLUX.1/Wan/HunyuanVideo/SD3 native paths, unavailable via `sglang serve`/the OpenAI server, and mutually exclusive with TeaCache — do not assume it composes with an arbitrary deployment. - `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is documented as a dev/debug override, not a production configuration mechanism. - `SGLANG_EXTERNAL_MODEL_PACKAGE` must be set before process startup; each process imports the package exactly once, and `overwrite=True` is required to intentionally replace a built-in pipeline. - A missing model ID in the compatibility matrix does not mean unsupported — check for detector-based aliases or a local directory match before concluding a family is unavailable. - HunyuanVideo/FastHunyuan default to tiled VAE decode for a reason: overriding `--vae-config.parallel-decode-mode` to `spatial`/`spatial_shard` at their documented shapes can request hundreds of GiB per rank for the causal mask — only use those modes at smaller validated shapes. ## Related Concepts - [[concepts/diffusion-optimization]] — the performance/acceleration stack (caching, attention backends, kernels, parallelism, disaggregation, batching, quantization, profiling) that the deployment and performance-mode levers described here build on. - [[concepts/sglang-overview]] — the LLM-serving core SGLang Diffusion ships alongside. - [[concepts/supported-models]] — SGLang's non-diffusion (LLM/multimodal/embedding/reward) model coverage. - [[concepts/quantization]] — SGLang's LLM-side quantization support, contrasted with diffusion checkpoint quantization in [[concepts/diffusion-optimization]]. - [[concepts/parallelism-and-disaggregation]] — SGLang's LLM-side TP/PP/EP/DP and PD disaggregation, contrasted with the diffusion-specific CFG/TP/SP parallelism and encoder/denoiser/decoder disaggregation in [[concepts/diffusion-optimization]]. - [[concepts/server-apis]] — the LLM-serving OpenAI/native/Anthropic/Ollama API surface, alongside which the diffusion `/v1/images` and `/v1/videos` endpoints are served. ## Sources - raw/github_doc-docs-docs-sglang-diffusion-index-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-installation-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-api-cli-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-api-openai-api-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-api-post-processing-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-models-with-ar-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-models-with-pe-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-realtime-models-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-spectrum-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-compatibility-matrix-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-deployment-cookbook-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-support-new-models-mdx.md - raw/github_doc-docs-docs-sglang-diffusion-environment-variables-mdx.md --- title: "Frontend DSL" type: concept tags: [frontend, api, foundational, well-established, user] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-references-frontend-frontend-index-mdx.md", "raw/github_doc-docs-docs-references-frontend-frontend-tutorial-mdx.md", "raw/github_doc-docs-docs-references-frontend-choices-methods-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition The SGLang **frontend language** is a Python-embedded DSL for defining structured, multi-step LLM programs — prompts with control flow, parallel branches, and constrained generation — as ordinary Python functions decorated with `@function`, rather than as raw strings sent to a chat-completions endpoint (raw/github_doc-docs-docs-references-frontend-frontend-tutorial-mdx.md). **Sourcing note:** this KB's `raw/github_doc-docs-docs-sglang-*.md` file set (29 files) does **not** cover this DSL — that entire set is SGLang Diffusion (image/video model) documentation. The frontend language is instead documented under `raw/github_doc-docs-docs-references-frontend-*.md` (`frontend_index.mdx`, `frontend_tutorial.mdx`, `choices_methods.mdx` — 3 files), which is what this page is grounded in. ## How It Works A frontend program is a Python function taking a program state `s` as its first argument, decorated with `@sgl.function` (or `@function`). Statements append to `s` with `+=`; `gen(name, ...)` triggers generation and stores the result under `state[name]`. **Basic usage:** ```python Example @function def basic_qa(s, question): s += system(f"You are a helpful assistant than can answer questions.") s += user(question) s += assistant(gen("answer", max_tokens=512)) ``` **Multi-turn dialog** simply chains more `user`/`assistant` turns onto the same state `s` before returning it. **Control flow** is plain Python — branch on a previously generated value: ```python Example s += assistant("... " + gen("tool", choices=["calculator", "search engine"]) + ". ") if s["tool"] == "calculator": s += assistant("The math expression is: " + gen("expression")) elif s["tool"] == "search engine": s += assistant("The key word to search is: " + gen("word")) ``` **Parallelism via `fork`.** `s.fork(n)` launches `n` parallel branches; because `sgl.gen` is non-blocking, a loop over forks issues concurrent generation calls that run in parallel: ```python Example forks = s.fork(2) for i, f in enumerate(forks): f += assistant(f"Now, expand tip {i+1} into a paragraph:\n" + gen("detailed_tip", max_tokens=256, stop="\n\n")) s += assistant("Tip 1:" + forks[0]["detailed_tip"] + "\n") ``` **Constrained decoding via `regex`.** `gen(..., regex=...)` constrains output to a pattern — a plain regex (e.g. an IP-address shape) or a hand-built regex encoding a JSON schema. This is documented as **only supported for local models** (raw/github_doc-docs-docs-references-frontend-frontend-tutorial-mdx.md). **Batching.** `function_name.run_batch([...dicts of kwargs...], progress_bar=True)` runs many prompt instances and returns a list of states. **Streaming.** `function_name.run(..., stream=True)` returns a state whose `.text_iter()` yields output incrementally. **Complex prompt structure.** `system(...)`, `with s.user(): ...`, and `assistant_begin()`/`assistant_end()` give explicit control over role-block boundaries beyond the simple `+=` shorthand. **Multi-modal generation.** `image(image_file)` can be concatenated into a `user(...)` turn for VLM prompts (server launched with a vision model, e.g. `Qwen2.5-VL-7B-Instruct`). **Backend selection.** `set_default_backend(RuntimeEndpoint(f"http://localhost:{port}"))` points frontend programs at a locally launched SGLang server; the doc notes OpenAI or other API endpoints can be used as alternate backends (raw/github_doc-docs-docs-references-frontend-frontend-tutorial-mdx.md). ### Choices methods The optional `choices_method` argument on `gen(..., choices=[...])` controls how SGLang scores and selects among a fixed set of options. It's only supported on the `RuntimeEndpoint` backend — other backends like OpenAI have their own bespoke selection logic due to API limitations (raw/github_doc-docs-docs-references-frontend-choices-methods-mdx.md). | Method | How it scores | Known failure mode | |---|---|---| | `token_length_normalized` (**default**) | Highest average logprob across all of an option's tokens | Performs poorly when one option has many tokens whose later tokens are predicted with high confidence given the earlier ones (e.g. `["Paris", "Antidisestablishmentarianism"]`) | | `greedy_token_selection` | Highest logprob for the option's *initial* token only (shorter overlapping options get their logprob extended by their average for fair comparison against longer ones) | Misled by an attractive initial token — e.g. asked to "Name a US president" with choices `["Donald Duck", "Millard Fillmore"]`, greedy selection picks the wrong one | | `unconditional_likelihood_normalized` | Average token logprob normalized by *unconditional* token logprobs (per an EleutherAI method) | Requires an extra LLM call to obtain the unconditional likelihoods | ## Key Parameters - `gen(name, max_tokens=..., stop=..., temperature=..., regex=..., choices=[...], choices_method=...)` — the core generation primitive. - `s.fork(n)` — parallel branch count for concurrent generation. - `choices_method` — one of `sgl.token_length_normalized` (default), `sgl.greedy_token_selection`, `sgl.unconditional_likelihood_normalized`; only honored by `RuntimeEndpoint`. - `regex` — decoding constraint; documented as only supported for local models per the tutorial doc. - `stream=True` on `.run(...)` for incremental output via `.text_iter()`. ## When To Use Reach for the frontend DSL when a task needs **structure the plain chat-completions API doesn't give you**: branching logic driven by an earlier generation, fan-out/fan-in parallel generation (`fork`), schema-constrained output built as a hand-rolled regex, or batch execution of the same templated program across many inputs. For a single request/response call, the OpenAI-compatible or native HTTP APIs (see `[[concepts/server-apis]]`) are simpler. ## Risks & Pitfalls - `choices_method` failure modes are real and example-documented (see table above) — the default (`token_length_normalized`) is not always the right choice for short-vs-long or misleading-prefix option sets. - `regex`-constrained decoding is stated as local-model-only; it won't work against a remote/OpenAI backend. - `choices_method` itself is not supported on non-`RuntimeEndpoint` backends — they substitute their own bespoke selection logic due to API limitations, so behavior can differ across backends for the same program. ## Related Concepts - `[[concepts/architecture-and-radixattention]]` — the continuous-batching scheduler that makes `fork`'s non-blocking parallel `gen` calls efficient - `[[concepts/server-apis]]` — the HTTP surface (`RuntimeEndpoint`) the frontend DSL talks to - `[[concepts/sampling-parameters]]` — `temperature`, `stop`, and related generation controls used inside `gen(...)` - `[[concepts/structured-outputs-and-tool-calling]]` (planned) — server-side grammar/JSON-schema constrained decoding, distinct from this DSL's client-side `regex` argument ## Sources - raw/github_doc-docs-docs-references-frontend-frontend-index-mdx.md - raw/github_doc-docs-docs-references-frontend-frontend-tutorial-mdx.md - raw/github_doc-docs-docs-references-frontend-choices-methods-mdx.md --- title: "Hierarchical Caching" type: concept tags: [caching, architecture, advanced, operator, well-established] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-hicache-mdx.md" - "raw/github_doc-docs-docs-advanced-features-hicache-design-mdx.md" - "raw/github_doc-docs-docs-advanced-features-hicache-best-practices-mdx.md" - "raw/github_doc-docs-docs-advanced-features-hicache-storage-runtime-attach-d.md" - "raw/github_doc-docs-docs-advanced-features-session-radix-cache-mdx.md" - "raw/github_doc-docs-docs-advanced-features-quantized-kv-cache-mdx.md" - "raw/github_doc-docs-docs-advanced-features-hisparse-guide-mdx.md" - "raw/github_doc-docs-docs-advanced-features-dcp-mdx.md" confidence: medium --- # Hierarchical Caching ## Definition SGLang extends its baseline RadixAttention prefix cache (GPU-only) with a family of mechanisms that widen, prioritize, or shrink the KV cache: **HiCache** organizes the KV cache into a three-tier hierarchy (GPU → host memory → distributed storage) so far more shared-prefix data can be cached and reused; **session-aware radix caching** biases eviction order to protect KV belonging to active multi-turn sessions; **quantized KV cache** shrinks the per-token memory footprint by storing KV in FP8 or FP4 instead of BF16; and **HiSparse** (a decode-side sparse-attention optimization for DeepSeek Sparse Attention / DeepSeek V4 models) keeps only a small "hot" KV window on GPU while the full KV lives in CPU pinned memory. All of these sit on top of, or alongside, [[concepts/architecture-and-radixattention]]'s core RadixTree. ## How It Works ### HiCache: three-tier KV hierarchy Modeled on CPU cache hierarchies: **L1** = GPU memory (private per instance), **L2** = host (CPU) memory (private per instance), **L3** = distributed storage (shared across all instances in a cluster). Metadata is tracked in a **HiRadixTree** — an extension of the RadixAttention RadixTree where each node records which tier(s) hold its KV span. L1/L2 locations are tracked precisely (exact storage address); L3 metadata is *not* synchronized locally — it's queried from the storage backend in real time to reduce overhead. Per-request workflow: **local match** (traverse HiRadixTree across L1+L2, page-granularity if `page_size > 1`, splitting nodes at partial-match boundaries — pure metadata walk, no data copy, so it's fast) → **prefetch from L3** for the unmatched remainder (only triggered once the L3 hit length exceeds a threshold, default 256 tokens) → prefill computation on GPU → **write-back** of newly generated KV into L2/L3. **Prefetch termination strategies** (`--hicache-storage-prefetch-policy`): - `best_effort` — stop as soon as GPU can start prefill; lowest latency. - `wait_complete` — block until all prefetch completes; highest cache-hit rate. - `timeout` — bounded wait, **recommended for production** (helps meet SLOs). Timeout = `min(prefetch_timeout_max, prefetch_timeout_base + prefetch_timeout_per_ki_token * num_token_to_fetch / 1024)`, defaults: `prefetch_timeout_base=2s`, `prefetch_timeout_per_ki_token=0.1s`, `prefetch_timeout_max=30s`. **Write-back policies** (`--hicache-write-policy`): - `write_through` — every access immediately propagated to the next tier; strongest caching benefit, most I/O. - `write_through_selective` — only backs up data once its access frequency exceeds a threshold; less I/O. - `write_back` — only written down on eviction from the tier above; best when storage capacity is constrained. Cross-instance sharing: when data moves L2→L3, only data not already in L3 is transferred; anything landing in L3 becomes visible to every SGLang instance sharing that backend (subject to the backend's own semantics). **Multi-rank synchronization**: under TP, `all_reduce(op=min)` is used twice per prefetch — once so every rank agrees on the number of L3 hits (avoiding inconsistent prefetch-threshold decisions), and once after prefetch completes/terminates so every rank agrees on the successfully retrieved prefix length. **Data transfer optimization**: - Zero-copy L2↔L3 transfers pass memory addresses/sizes directly. - Memory layouts (`--hicache-mem-layout`): `layer_first` (default, matches GPU's natural layer-by-layer computation), `page_first` (contiguous per-page storage, best I/O efficiency, zero-copy to L3, but requires per-token-per-layer transfer to GPU), `page_first_direct` (groups all tokens of one layer within a page so L2→GPU transfers aggregate at page-layer granularity — same zero-copy perf as `page_first` but GPU-transfer-friendly, and compatible with the `fa3` attention backend). - CPU→GPU compute-transfer overlap: loads layer N+1's KV while computing layer N. - GPU-assisted I/O kernels (`--hicache-io-backend kernel`) beat plain `cudaMemcpyAsync` (`direct`) by up to 3x transfer speed. - MLA write-back optimization: since every TP rank holds identical full KV for MLA models (unlike MHA/GQA where each rank holds `1/tp_size`), only one rank initiates the write-back to avoid redundant storage. **Storage backends** (`--hicache-storage-backend {file,mooncake,hf3fs,nixl,aibrix,dynamic}`): Mooncake (RDMA, multi-NIC, zero-copy), DeepSeek 3FS / HF3FS (Kubernetes-native distributed storage), NIXL (unified API over 3FS, GPU Direct Storage, S3-compatible object storage), AIBrix KVCache (production KVCache-offload framework), HiCacheFile (simple file-based, demo only). All implement `class HiCacheStorage(ABC)` with `get`/`exists`/`set`. **LMCache** is a separate, alternative hierarchical-cache solution (not itself HiCache) also integrated into SGLang. A `dynamic` backend loads a custom class at runtime via `--hicache-storage-backend-extra-config '{"backend_name":..., "module_path":..., "class_name":...}'` (add `"interface_v1": 1` to use batch_get_v1/batch_set_v1). **Heterogeneous TP support**: HiCache storage supports cross-cluster KV reuse when deployments use different TP sizes but share the same storage namespace, via `tp_lcm_size` in `--hicache-storage-backend-extra-config` (set to the LCM of all sharing TP sizes, e.g. `{"tp_lcm_size": 8}` for TP∈{4,8}); for MHA models with Mooncake + `page_head` layout, HiCache splits head shards accordingly. **Runtime attach/detach (no restart)**: L3 backend can be swapped live via HTTP admin endpoints `PUT/DELETE/GET /hicache/storage-backend`, routed HTTP Server → TokenizerManager → Scheduler (strict idle check via `is_fully_idle()` — no running/waiting/queued requests of any kind) → `HiRadixCache.attach_storage_backend()`/`detach_storage_backend()` → `HiCacheController` (creates/destroys the backend instance via `StorageBackendFactory`, starts/stops its prefetch/backup threads). Non-idle attempts fail fast with HTTP 400 and make no state change. Under `dp_size > 1`, the request fans out to all DP scheduler ranks and only reports overall success if **every** rank succeeds — there is no automatic partial rollback (best practice: keep backend config identical across ranks; on failure, best-effort detach, fix config, retry). Detach only stops SGLang from *using* L3 — it does not delete data already stored remotely. **Integration with PD disaggregation** (see [[concepts/parallelism-and-disaggregation]]): HiCache can run on prefill nodes only (cross-prefill KV sharing, good for shared system prompts) or on both prefill and decode nodes with `--disaggregation-decode-enable-offload-kvcache` on the decode side (async offload so prefill nodes can reuse decode-generated KV in multi-turn dialogue). Uses the Mooncake TransferEngine for the underlying PD transport. ### Session-aware radix cache Multi-turn / long-lived sessions can get evicted under memory pressure just like any other cache entry. Session-aware caching (implemented only in `UnifiedRadixCache`) registers a request's reusable KV under a `session_id` so unreferenced KV is evicted **before** KV still referenced by an active session — this is soft protection (referenced KV *can* still be evicted if reclaiming unreferenced KV alone isn't enough), not a memory pin. Enable: ```bash SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 python3 -m sglang.launch_server --model-path MODEL_PATH --enable-session-radix-cache ``` Pass the same `session_id` on every request in a session (each request still needs the full intended prompt — the session ID only labels cache references, it does not reconstruct context): ```bash curl http://localhost:30000/generate -H "Content-Type: application/json" \ -d '{"text": "FULL_PROMPT_FOR_THIS_TURN", "sampling_params": {"max_new_tokens": 128}, "session_id": "agent-42"}' ``` No `/open_session` call is needed — registration happens automatically when a request finishes. Call `/close_session` (including on error/cancellation paths) to remove the session's references; this doesn't immediately free the KV, it just returns it to normal eviction order: ```bash curl -X POST http://localhost:30000/close_session -H "Content-Type: application/json" -d '{"session_id": "agent-42"}' ``` Eviction order by component: **Full attention** — unreferenced nodes first, then referenced nodes with fewer session references, then the configured policy (e.g. LRU). **SWA (sliding-window attention)** and **Mamba** — two LRU passes (unreferenced first, then referenced if more space is needed), scoped to the reusable tail/state on the registered leaf. Cascade rule: evicting an internal Full node also evicts its SWA and Mamba data; evicting SWA also evicts Mamba data; evicting Mamba affects only Mamba; evicting a leaf removes all component data on it. ### Quantized KV cache Stores KV pairs in lower precision than the model's compute dtype (BF16) to fit more tokens in the same memory. Set with `--kv-cache-dtype`: - `fp8_e5m2` — 5 exponent/2 mantissa bits, larger dynamic range (±57344.0), lower precision. - `fp8_e4m3` — 4 exponent/3 mantissa bits, higher precision, smaller range (±240.0). **Recommended default** for accuracy. - `nvfp4` / `fp4_mx_block16` — experimental MXFP4 (E2M1, 1 sign/2 exp/1 mantissa bit), block-based microscaling (SGLang uses 16-element blocks vs. OCP's 32). ```bash python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-0528 --kv-cache-dtype fp8_e4m3 ``` FP8 scaling factors: per-tensor (scalar) only, currently. Loaded automatically from pre-quantized checkpoints (e.g. ModelOpt `k_scale`/`v_scale`) or supplied via `--quantization-param-path` pointing to a JSON of `{"kv_cache": {"dtype": ..., "scaling_factor": {tp_rank: {layer_idx: value}}}}`. If no scaling factor is found anywhere, it silently defaults to `1.0`, which can cause accuracy issues. FP4 computes its block-based scaling factors automatically/dynamically — no external file needed. Memory savings: FP4 (block-16) supports ~1.78x more tokens than FP8, and ~3.56x more than BF16. Accuracy impact: FP8 E4M3 is typically near-lossless; FP4 holds up well on large models (200B+ params) on simple datasets, though the direction and size of the effect is model-specific — in preliminary accuracy tests (PR #10078 for MLA, PR #12612 for MHA) on gsm8k, Qwen3-235B-A22B actually *gains* ~0.18pp with FP4 vs. BF16 (0.9186 vs. 0.9168), while DeepSeek-R1-0528 *loses* ~0.33pp vs. BF16 and ~0.30pp vs. FP8 (0.9124 vs. 0.9157/0.9154) — but FP4 degrades more on smaller models (GPT-OSS-120B on aime25 drops from 0.7667 at FP8 to 0.3533 at FP4) and on harder/longer-context tasks. These are preliminary, first-party results; the source recommends evaluating FP4 accuracy on the specific model and workload rather than treating this table as settled/final behavior. **Critical caveat**: if the chosen attention backend cannot fuse KV dequantization into the attention kernel, throughput can regress badly enough to erase the memory win — always verify backend support (see [[concepts/attention-backends-and-cuda-graph]]). ### HiSparse: hierarchical sparse attention (decode-side GPU/host KV split) For models using **DeepSeek Sparse Attention (DSA)** (DeepSeek-V3.2, GLM-5.1) or **DeepSeek V4**, which natively select only a subset of tokens for attention, HiSparse keeps just a small fixed-size "hot" KV buffer on GPU per request (e.g. 4K token slots) while the complete KV lives in CPU pinned memory — without accuracy loss, because these architectures already only attend to a top-k token subset. This is distinct from HiCache: HiCache widens the *shared-prefix* cache hierarchy; HiSparse shrinks the *per-request decode* GPU footprint for sparse-attention models. HiSparse **requires PD disaggregation mode** and runs only on the **decode instance** — the prefill instance is unaware of it. Per-decode-step flow: forward decode → top-k selection (via attention scores) → swap-in (CUDA kernel loads top-k KV host→device; short sequences hit a fast path, long sequences do hit-detection/LRU-reorder/miss-handling) → decode attention on the top-k device locations → eager async backup of the previous token's KV device→host. In PD mode, the prefill instance RDMAs KV directly into the decode instance's **host** pool (bypassing decode GPU entirely), eliminating the transient GPU memory spike from a staged transfer. For DeepSeek V4, only C4 KV goes through this direct-to-host path; the c4_indexer and C128 KV remain device-to-device. Flags: `--enable-hisparse` (decode instance only) and `--hisparse-config` JSON: `top_k` (topk entries), `device_buffer_size` (GPU token-slot count), `host_to_device_ratio` (host:device pool size ratio — recommended `5` for ~1TB host memory, `10` for ~2TB), `swap_in_block_size` (CUDA thread-block size, default 960). Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'`. A **shared-index prefetch** optimization is automatic for eligible models (no pipeline parallelism, no speculative decoding): when a model reuses one anchor layer's top-k selection across subsequent "skip" layers (DSA `index_topk_freq`/`index_topk_pattern`; native as IndexShare in GLM-5.2), HiSparse replays the anchor's miss plan on a side stream so skip-layer host→device IO overlaps intervening compute. Disable for A/B testing with `SGLANG_DISABLE_HISPARSE_PREFETCH=1`. DSA decode backend auto-selects by `--kv-cache-dtype`: `bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv` (except GLM DSA on SM120/SM121 with fp8_e4m3, which uses `flashinfer_sparse_mla` — the only DSA kernel on that architecture). DeepSeek V4 always uses its own `dsv4` attention backend regardless of these flags. Example deployment (prefill and decode as separate processes, PD disaggregation required): ```bash # Prefill instance — no HiSparse flags needed python3 -m sglang.launch_server --model-path /path/to/model --trust-remote-code \ --port 8000 --host 0.0.0.0 --context-length 81920 --chunked-prefill-size 65536 \ --tp-size 8 --dp-size 8 --enable-dp-attention --mem-fraction-static 0.85 \ --disaggregation-mode prefill --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \ --nnodes 1 --node-rank 0 # Decode instance — with HiSparse python3 -m sglang.launch_server --model-path /path/to/model --trust-remote-code \ --port 8000 --host 0.0.0.0 --context-length 81920 --tp-size 8 --dp-size 8 --enable-dp-attention \ --mem-fraction-static 0.85 --disable-radix-cache --disaggregation-mode decode \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --dist-init-addr 127.0.0.1:5757 \ --nnodes 1 --node-rank 0 --enable-hisparse \ --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}' ``` ## Key Parameters - `--enable-hierarchical-cache`, `--hicache-ratio` (host:GPU pool size ratio, must be `>1`), `--hicache-size` (host pool GB **per rank**, overrides `--hicache-ratio` if set), `--page-size` — HiCache core sizing. - `--hicache-storage-backend {file,mooncake,hf3fs,nixl,aibrix,dynamic}`, `--hicache-storage-backend-extra-config` — L3 backend selection/config. - `--hicache-storage-prefetch-policy {best_effort,wait_complete,timeout}`, `--hicache-write-policy {write_back,write_through,write_through_selective}` — HiCache runtime behavior. - `--hicache-io-backend {direct,kernel}`, `--hicache-mem-layout {layer_first,page_first,page_first_direct}` — HiCache data-movement tuning. - `--enable-lmcache`, `--lmcache-config-file` — alternative to HiCache. - `SGLANG_ENABLE_UNIFIED_RADIX_TREE=1`, `--enable-session-radix-cache`, `session_id` request field — session-aware caching. - `--kv-cache-dtype {fp8_e5m2,fp8_e4m3,nvfp4,fp4_mx_block16}`, `--quantization-param-path` — quantized KV cache. - `--enable-hisparse`, `--hisparse-config` (`top_k`, `device_buffer_size`, `host_to_device_ratio`, `swap_in_block_size`), `SGLANG_DISABLE_HISPARSE_PREFETCH` — HiSparse. ## When To Use - **HiCache**: long-context or multi-turn/multi-QA workloads where GPU-only prefix caching runs out of capacity — especially when repeated system prompts or shared prefixes span more data than fits in GPU memory. - **HiCache + PD disaggregation**: system-prompt-heavy workloads (prefill-only HiCache) or multi-turn conversations spanning prefill/decode boundaries (full HiCache with decode-side async offload). - **Session-aware radix cache**: agentic or chat workloads with many concurrent long-lived sessions competing for cache space under memory pressure. - **Quantized KV cache**: throughput/context-length-bound deployments willing to trade a small, format-dependent accuracy cost for significantly more cacheable tokens — verify the attention backend fuses dequantization first. - **HiSparse**: long-context decode serving of DSA/DeepSeek-V4 models where GPU KV memory per request is the concurrency bottleneck; requires committing to PD disaggregation mode. ## Risks & Pitfalls - Larger HiCache size does not scale hit rate linearly — once hot/reusable data is already cached, further size increases yield only marginal gains; tune to workload, not by defaulting to "bigger." - `page_first` layout silently falls back to `layer_first` if the I/O backend is `direct` (only compatible with `kernel`) — a layout/backend mismatch won't error, it will just stop being optimized as configured. - Runtime storage-backend attach/detach requires the server to be **fully idle** (no running or queued requests anywhere, including disaggregation queues) — attempts under load fail fast (HTTP 400) with no state change; under `dp_size > 1`, a failure can mean some DP ranks already succeeded with no automatic rollback. - Detaching a storage backend does not delete already-written remote data in Mooncake/HF3FS/etc. - Quantized KV cache without backend-fused dequantization can be *slower* than BF16 despite using less memory — always check attention-backend support (see [[concepts/attention-backends-and-cuda-graph]]) before enabling. - FP4 KV cache is experimental and accuracy-sensitive on smaller models and harder/longer-context tasks; missing scaling factors default silently to `1.0` and can degrade accuracy without warning. - Session-aware cache references are soft protection only — referenced KV can still be evicted under sufficient memory pressure; don't rely on it as a hard pin. - DCP × HiCache L2 composition (see [[concepts/parallelism-and-disaggregation]]) currently supports only MLA L1/L2 — L3, LMCache, HiSparse, non-MLA host pools, speculative decoding, and PD decode are all unsupported in that specific combination. - HiSparse is decode-instance-only and hard-requires PD disaggregation — it cannot be used in a unified (non-disaggregated) deployment. ## Related Concepts - [[concepts/architecture-and-radixattention]] — the RadixAttention/RadixTree foundation that HiRadixTree and UnifiedRadixCache extend. - [[concepts/parallelism-and-disaggregation]] — PD disaggregation is a prerequisite for HiSparse and integrates with HiCache's prefill/decode split; DCP composes narrowly with HiCache L2. - [[concepts/attention-backends-and-cuda-graph]] — attention backend choice determines quantized-KV-cache and DSA-sparse-decode kernel support (`flashmla_sparse`, `flashmla_kv`, `flashinfer_sparse_mla`, `dsv4`). - [[concepts/quantization]] — model-weight quantization is a separate axis from KV-cache quantization, though both reduce memory. - [[concepts/observability-and-determinism]] — `--enable-metrics`/`--enable-cache-report` expose HiCache hit-rate and prefetch metrics. ## Sources - raw/github_doc-docs-docs-advanced-features-hicache-mdx.md - raw/github_doc-docs-docs-advanced-features-hicache-design-mdx.md - raw/github_doc-docs-docs-advanced-features-hicache-best-practices-mdx.md - raw/github_doc-docs-docs-advanced-features-hicache-storage-runtime-attach-d.md - raw/github_doc-docs-docs-advanced-features-session-radix-cache-mdx.md - raw/github_doc-docs-docs-advanced-features-quantized-kv-cache-mdx.md - raw/github_doc-docs-docs-advanced-features-hisparse-guide-mdx.md - raw/github_doc-docs-docs-advanced-features-dcp-mdx.md --- title: "Installation" type: concept tags: [install, foundational, user, operator] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-get-started-install-mdx.md", "raw/github_doc-docs-docs-get-started-quickstart-mdx.md", "raw/github_doc-readme-md.md"] confidence: high sglang_version: "v0.5.18" --- ## Definition Installing SGLang means getting the `sglang` Python package (and its compiled kernel dependencies) onto a machine and launching an inference server process, `sglang.launch_server`, that loads a model and serves it. SGLang documents seven installation paths, primarily targeting common NVIDIA GPU platforms, with dedicated pages for AMD GPUs, Apple Metal, Intel Xeon CPUs, Google TPU, NVIDIA DGX Spark, NVIDIA Jetson, Ascend NPUs, and Intel XPU (raw/github_doc-docs-docs-get-started-install-mdx.md). ## How It Works **Prerequisite:** Python 3.10 or higher (raw/github_doc-docs-docs-get-started-install-mdx.md). ### Method 1 — pip / uv (recommended) ```bash pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` `--prerelease=allow` is required because some of SGLang's dependencies only publish pre-releases on PyPI; without it, uv older than 0.12.0 silently installs an older SGLang (0.5.9). uv 0.12.0+ treats the flag as a harmless no-op. The default CUDA major version is 13. To install under CUDA 12: ```bash pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang uv pip install --force-reinstall torch==2.13.0 torchaudio==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu129 uv pip install --force-reinstall sglang-kernel --index-url https://docs.sglang.ai/whl/cu129/ uv pip install --force-reinstall sgl-deep-gemm --index-url https://docs.sglang.ai/whl/cu129/ --no-deps ``` **Nightly builds** are published from the latest `main` to a dedicated wheel index; add it with `--extra-index-url` and combine `--prerelease=allow` with `--index-strategy unsafe-best-match`: ```bash pip install --upgrade pip pip install uv uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang ``` (swap `cu130` for `cu129` under CUDA 12.) ### Method 2 — from source ```bash git clone -b v0.5.18 https://github.com/sgl-project/sglang.git cd sglang pip install --upgrade pip pip install -e "python" ``` For development, you can try the `lmsysorg/sglang:dev` docker image (raw/github_doc-docs-docs-get-started-install-mdx.md). ### Method 3 — Docker Images are published to Docker Hub at `lmsysorg/sglang`. `latest` and `dev` are **mutable** tags (`latest` tracks the newest stable release; `dev` rebuilds daily from `main`); pin an immutable version tag (e.g. `lmsysorg/sglang:v0.5.18`) for reproducible deployments. ```bash docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 ``` For production, the `-runtime` variant is ~40% smaller (excludes build tools and dev dependencies) — e.g. `lmsysorg/sglang:latest-runtime`. SGLang ships with a CUDA 13 environment by default; use images with `-cu12`/`-cu129` suffixes for CUDA 12. ### Method 4 — Kubernetes Use [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade LLM management, or apply the raw manifests directly: `docker/k8s-sglang-service.yaml` for single-node serving, `docker/k8s-sglang-distributed-sts.yaml` for multi-node serving of large models (e.g. DeepSeek-R1). ### Method 5 — Docker Compose Copy `docker/compose.yaml` and run `docker compose up -d`. The docs note the k8s service manifest is a better approach for serving as a long-running service. ### Method 6 — SkyPilot Install SkyPilot, then deploy to any of 12+ clouds or a Kubernetes cluster with a single command (`sky launch -c sglang --env HF_TOKEN sglang.yaml`), retrieving the HTTP endpoint via `sky status --endpoint 30000 sglang`. SkyServe adds autoscaling and failure recovery on top. ### Method 7 — AWS SageMaker Deploy a pre-built SGLang Deep Learning Container, or build a custom container from `docker/sagemaker.Dockerfile` and the `docker/serve` script, push to ECR, and deploy per `examples/sagemaker/deploy_and_serve_endpoint.py`. The default SageMaker server command is `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`; serving parameters are passed as `SM_SGLANG_*` environment variables, which the `serve` script converts into `--*` CLI flags (e.g. `SM_SGLANG_REASONING_PARSER=qwen3` becomes `--reasoning-parser qwen3`). ### Launching the server Across every method, the actual serving process is the same command, e.g.: ```bash python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000 ``` Wait for `The server is fired up and ready to roll!` in the terminal output before sending requests (raw/github_doc-docs-docs-get-started-quickstart-mdx.md). Once running, API docs are auto-served at `/docs` (Swagger UI), `/redoc` (ReDoc), and `/openapi.json` (OpenAPI spec). The server automatically applies the chat template from the Hugging Face tokenizer; override it with `--chat-template` if needed. ## Key Parameters - `--model-path` — the Hugging Face model id or local path to serve. - `--host` / `--port` — bind address (commonly `0.0.0.0` and `30000`). - `--chat-template` — override the auto-detected chat template. - `CUDA_HOME` — if unset, causes `OSError: CUDA_HOME environment variable is not set`; fix with `export CUDA_HOME=/usr/local/cuda-` or by installing FlashInfer first. - `HF_TOKEN` — Hugging Face token, required as an env var for Docker/SageMaker deployments of gated models. ## When To Use - **pip/uv**: the default path for most users on common NVIDIA GPUs. - **From source**: contributing to SGLang or needing an unreleased fix; use the `dev` docker image if doing active development. - **Docker**: reproducible, portable deployment; pin an immutable version tag rather than `latest`/`dev`. - **Kubernetes / SkyPilot / SageMaker**: production or multi-node deployments needing orchestration, autoscaling, or managed cloud infrastructure. - **Nightly builds**: to pick up unreleased features/fixes ahead of the next stable release, at the cost of stability. ## Risks & Pitfalls - Without `--prerelease=allow`, uv older than 0.12.0 silently installs a stale SGLang version (0.5.9) instead of failing loudly — a silent-downgrade trap. - `latest` and `dev` Docker tags are mutable and get overwritten over time; using them for production deployments breaks reproducibility. Pin an immutable version tag instead. - FlashInfer is the default attention kernel backend and only supports sm75 and above; on affected devices (T4, A10, A100, L4, L40S, H100) FlashInfer-related issues are worked around with `--attention-backend triton --sampling-backend pytorch` (raw/github_doc-docs-docs-get-started-install-mdx.md). See [[concepts/attention-backends-and-cuda-graph]] (planned). - `OSError: CUDA_HOME environment variable is not set` is a common first-run failure; two fixes are documented (set `CUDA_HOME` manually, or install FlashInfer first per its own installation doc). - CUDA major-version mismatch: the default install assumes CUDA 13; running under CUDA 12 requires extra `--force-reinstall` steps for torch/sglang-kernel/sgl-deep-gemm with the `cu129` wheel index, and Docker images need the `-cu12`/`-cu129` suffix. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/sending-requests]] - [[concepts/offline-engine]] - [[concepts/server-arguments]] (planned) - [[concepts/supported-hardware]] (planned) - [[concepts/attention-backends-and-cuda-graph]] (planned) ## Sources - raw/github_doc-docs-docs-get-started-install-mdx.md - raw/github_doc-docs-docs-get-started-quickstart-mdx.md - raw/github_doc-readme-md.md --- title: "LoRA and Model Loading" type: concept tags: [lora, api, advanced, operator, well-established] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-lora-mdx.md" - "raw/github_doc-docs-docs-advanced-features-model-loading-mdx.md" - "raw/github_doc-docs-docs-advanced-features-object-storage-mdx.md" - "raw/github_doc-docs-docs-advanced-features-checkpoint-engine-mdx.md" - "raw/github_doc-docs-docs-advanced-features-rfork-mdx.md" confidence: medium --- # LoRA and Model Loading ## Definition Two related lifecycle concerns for getting a served model onto GPUs: **multi-LoRA serving** (running many LoRA adapters concurrently against one base model, dynamically swapping which adapter handles each request in a batch) and **model weight loading** (how SGLang reads base-model weights into memory — from local disk, object storage, another running instance, or a distributed checkpoint-loading system). Both share the same underlying goal of minimizing GPU idle time during initialization or adapter switching. ## How It Works ### LoRA serving SGLang serves multiple LoRA adapters against one base model using techniques from **S-LoRA** and **Punica**, letting different sequences within the same batch use different adapters (or the base model, via `None`). Core flags: | Flag | Meaning | Default | |---|---|---| | `--enable-lora` | Enable LoRA support (auto-set if `--lora-paths` given) | off | | `--lora-paths` | Adapters to preload: ``, `=`, or JSON `{"lora_name":str,"lora_path":str,"pinned":bool}` | — | | `--max-loras-per-batch` | Max adapters used per batch (drives GPU memory reservation) | `8` | | `--max-loaded-loras` | Cap on adapters resident in **CPU** memory at once (must be ≥ `max-loras-per-batch`; when overlap loading is enabled, capped at ≤ 2× `max-loras-per-batch`) | — | | `--lora-eviction-policy` | `lru` (default, better cache efficiency) or `fifo` | `lru` | | `--lora-backend` | GEMM kernel backend: `triton` or `csgmv` (chunked SGMV) | `csgmv` | | `--max-lora-rank` | Max supported LoRA rank; auto-inferred from `--lora-paths` if unset — **must** be set explicitly if you plan to dynamically load larger-rank adapters later | auto | | `--lora-target-modules` | Union of modules LoRA applies to (e.g. `q_proj`); `all` enables every supported module at a minor performance cost; auto-inferred from `--lora-paths` if unset — set explicitly for dynamic loading of adapters with different target modules | auto | | `--max-lora-chunk-size` | Max chunk size for the `csgmv` backend; larger may improve performance, tune per hardware/workload | `16` | | `--lora-drain-wait-threshold` | Seconds an adapter's request can wait before the scheduler selectively drains a running adapter to make room (mitigates tail latency under skewed load); `0` disables | `0` | | `--enable-lora-overlap-loading` | Overlap H2D adapter-weight transfer with GPU compute | off | TP is supported for LoRA serving (sharding strategy per the S-LoRA paper). **Per-request usage**: native `/generate` API takes a `lora_path` list parallel to the `text`/prompt list (`None` = base model for that sequence): ```python json_data = { "text": ["List 3 countries and their capitals.", "List 3 countries and their capitals."], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, "lora_path": ["lora0", None], } ``` OpenAI-compatible API uses `model:adapter-name` syntax (e.g. `base-model:adapter_a`) in the `model` field of `/v1/chat/completions` or `/v1/completions`. **Dynamic loading**: `/load_lora_adapter` (`{"lora_name": ..., "lora_path": ...}`) and `/unload_lora_adapter` (`{"lora_name": ...}`) HTTP endpoints add/remove adapters without a restart. When relying on dynamic loading, explicitly set `--max-lora-rank` and `--lora-target-modules` at startup — otherwise SGLang infers them from `--lora-paths` and every dynamically loaded adapter must then be the same shape or "strictly smaller" than the initial set. **GPU pinning**: mark an adapter `"pinned": true` (in `--lora-paths` JSON or the `/load_lora_adapter` body) to keep it permanently resident in one of the `--max-loras-per-batch` GPU slots, avoiding repeated H2D transfer/reinit for hot adapters. Trade-off: fewer free slots for dynamic adapters; SGLang caps pinned adapters at `max-loras-per-batch - 1` to prevent unpinned requests from being starved entirely. **Backend choice**: `csgmv` (chunked SGMV, current default, optimized for high concurrency) is upstream-reported (2026-08-24 docs snapshot) at a 20–80% latency improvement over the basic `triton` backend — a workload/hardware-specific result, not a general guarantee. Future backends (Cutlass/CUDA-based) are planned. **Overlap loading** (`--enable-lora-overlap-loading`): hides adapter H2D transfer behind prefill/decode compute; upstream-reported at ~35% median TTFT reduction under adversarial conditions (a specific benchmarked scenario, not a general guarantee). Two caveats: (1) requires LoRA weights pinned in **CPU** memory for async H2D, so SGLang caps `max_loaded_loras ≤ 2 × max_loras_per_batch` when this is enabled; (2) reduces the scheduler's ability to batch multiple adapters into one prefill batch (adapters become GPU-ready at staggered times), which can **increase** TTFT when adapter load time is small relative to prefill compute — e.g. 4 adapters at 2ms load / 20ms prefill each: baseline (load all synchronously, one combined batch) ≈ `2×4 + 20 = 28ms`; with overlap loading and no cross-adapter batching, worst case ≈ `2 + 4×20 = 82ms`. This is why overlap loading is **off by default** — enable only after confirming adapter loading (not prefill compute) is the actual bottleneck (high adapter churn, heavy adapters, PCIe-bound). Roadmap items still in development: Embedding Layer LoRA, Unified Paging, Cutlass backend (tracked in sgl-project/sglang#2929). ### Model weight loading `--load-format` selects the loader, with auto-detection overriding `auto` for certain paths: Mistral native checkpoints → `mistral`; `.gguf` paths → `gguf`; object storage URIs (`s3://`, `gs://`, `az://`) → `runai_streamer`; remote URIs → `remote`. | Format | Behavior | |---|---| | `auto` (default) | `safetensors` if available, else PyTorch `.bin` | | `safetensors` / `pt` | Explicit format | | `npcache` | PyTorch `.bin` + numpy cache for faster subsequent loads (`.bin` only) | | `dummy` | Random weights, for profiling | | `sharded_state` | Each TP worker reads only its own pre-sharded shard (fast path for large TP models); build with `examples/runtime/engine/save_sharded_state.py` | | `fastsafetensors` | safetensors via the `fastsafetensors` iterator (GPUDirect Storage support) | | `layered` | Loads layer-by-layer so a layer can be quantized before the next loads — lowers peak memory | | `gguf` | GGUF format (auto-detected from `.gguf` path) | | `bitsandbytes` | bitsandbytes-quantized loading | | `mistral` | Mistral native format (auto-detected) | | `flash_rl` | Loads a BF16/FP16 checkpoint with native SGLang FP8 quantization for RL training; requires `--rl-quant-profile` | | `runai_streamer` | Streams from SSD/shared filesystem/object storage (see below) | | `remote` | Remote KV/filesystem connector (auto-detected for remote URIs) | | `remote_instance` | Pulls weights over the network from another running SGLang "seed" instance — this is **R-Fork** (below) | `--model-loader-extra-config` passes a JSON string of loader-specific options, e.g. `{"enable_multithread_load": true, "num_threads": 16}`. Key per-format options: - `auto`/`safetensors`/`pt`/`npcache`: `enable_multithread_load` (bool, default `true`; auto-disabled when `--weight-loader-prefetch-checkpoints` is set to avoid I/O oversubscription — opt back in explicitly if desired, e.g. on local NVMe where prefetch is a no-op), `num_threads` (default `8`). - `sharded_state`: `pattern` (default `model-rank-{rank}-part-{part}.safetensors`). - `fastsafetensors`: `enable_gds` (default `true`; set `false` where the host lacks the NVIDIA GPUDirect Storage kernel driver, e.g. gVisor sandboxes). - `bitsandbytes`: `qlora_adapter_name_or_path` — apply a QLoRA adapter on top of bitsandbytes-quantized base weights. - `runai_streamer`: `distributed`, `concurrency`, `memory_limit` (see Object Storage below). **Weight-loading performance flags** (independent of `--load-format`): `--download-dir` (HF cache location), `--weight-loader-disable-mmap` (helps on filesystems where mmap is slow), `--weight-loader-prefetch-checkpoints` (prefetch shards into OS page cache before loading; each rank prefetches a fraction, cutting shared-filesystem (NFS/Lustre) network I/O from N×checkpoint down to 1×checkpoint — recommended for network-storage models), `--weight-loader-prefetch-num-threads` (default `4`), `--weight-loader-drop-cache-after-load` (calls `posix_fadvise(DONTNEED)` per shard after loading to free page cache; supported by standard safetensors and `fastsafetensors` loaders), `--custom-weight-loader` (import path to a custom loading function, e.g. `my_package.weight_load_func`). ### Loading from object storage `runai_streamer` streams weights directly from S3 (`s3://`), GCS (`gs://`), Azure Blob (`az://`), or S3-compatible storage, without a full local download — auto-detected from the URI so `--load-format` can usually be omitted. Two-phase approach: metadata (config/tokenizer files) downloaded to local cache once before process launch; weights streamed lazily during model loading. ```bash python -m sglang.launch_server --model-path s3://my-bucket/models/llama-3-8b/ # with TP: python -m sglang.launch_server --model-path gs://my-bucket/models/llama-70b/ --tp 4 \ --model-loader-extra-config '{"distributed": true}' ``` Config params (via `--model-loader-extra-config`): `distributed` (bool, auto-`true` for object storage on CUDA-like devices — parallelizes streaming across TP processes), `concurrency` (int, default `4`, concurrent download streams — raise for large models), `memory_limit` (bytes, streaming buffer cap, system-dependent default). Limitations: only `.safetensors` is supported; distributed streaming requires CUDA-like devices (otherwise falls back to non-distributed). ### Checkpoint Engine integration A distributed weight-loading system (from Moonshot AI's `checkpoint-engine`, `pip install 'checkpoint-engine[p2p]'`) that parallelizes loading across processes/nodes and overlaps it with other init work (e.g. CUDA graph capture) — targeted at large models and multi-node deployments. Two components: the **SGLang server**, launched with `--load-format dummy --wait-for-initial-weights` (dummy weights let it proceed to CUDA graph capture etc. while waiting), and separate **checkpoint engine workers** (managed by `torchrun` or `python -m sglang.srt.checkpoint_engine.update`) that load and distribute real weights via a parameter-server architecture with three update modes: `broadcast` (loading processes broadcast to inference processes), `p2p` (direct peer-to-peer), `all` (both). ```bash # Terminal 1: server python -m sglang.launch_server --model-path Qwen/Qwen3-8B --tp 8 --load-format dummy --wait-for-initial-weights # Terminal 2: checkpoint engine python -m sglang.srt.checkpoint_engine.update --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ --inference-parallel-size 8 ``` Multi-node scales `--inference-parallel-size` to the aggregate TP across nodes (e.g. 16 for 2 nodes × TP8) and each server additionally sets `--dist-init-addr`, `--nnodes`, `--node-rank`. Key checkpoint-engine flags: `--update-method`, `--checkpoint-path`, `--inference-parallel-size`, `--endpoint` (default `http://localhost:19730`), `--checkpoint-name`, `--save-metas-file`/`--load-metas-file`, `--uds`, `--weight-version`. Performance: multi-node loading lets each node read only a portion of weights from disk, effectively multiplying disk bandwidth (preliminary test: ~20s faster loading DeepSeek-R1 on 2×H20-3e nodes); single-process `dummy`-format loading overlaps disk→CPU transfer with CUDA graph capture for additional savings. ### R-Fork (remote-instance weight loading) R-Fork (Tensor Remote Fork) loads weights via zero-copy GPU-to-GPU transfer from a running "seed" SGLang instance to a new "client" instance, upstream-reported as cutting boot time from minutes to seconds — set via `--load-format remote_instance`. Backends (`--remote-instance-weight-loader-backend`, default `nccl`): - **`nccl`**: seed launches normally; client adds `--load-format remote_instance --remote-instance-weight-loader-seed-instance-ip --remote-instance-weight-loader-seed-instance-service-port --remote-instance-weight-loader-send-weights-group-ports --remote-instance-weight-loader-backend nccl`. - **`transfer_engine`**: seed adds `--remote-instance-weight-loader-start-seed-via-transfer-engine`; client uses the same `remote_instance` flags with `--remote-instance-weight-loader-backend transfer_engine` (no port-list needed). - **`modelexpress`**: uses [ModelExpress](https://github.com/ai-dynamo/modelexpress), a coordination service managing P2P weight-transfer metadata via a centralized registry (removes the need for direct seed IP/port config) — requires a running ModelExpress server and the ModelExpress Python package in the SGLang image. Configure via `--modelexpress-config '{"url": "...", "transport": "nixl"|"transfer_engine"}'` (transport defaults to `nixl`; every instance uses the same command shape — if no ready source exists it loads natively and publishes to ModelExpress, otherwise it pulls via ModelExpress P2P transfer). ## Key Parameters - `--enable-lora`, `--lora-paths`, `--max-loras-per-batch`, `--max-loaded-loras`, `--lora-backend`, `--max-lora-rank`, `--lora-target-modules`, `--enable-lora-overlap-loading` — LoRA serving core. - `--load-format`, `--model-loader-extra-config` — model loader selection and per-loader config. - `--weight-loader-prefetch-checkpoints`, `--weight-loader-disable-mmap`, `--weight-loader-drop-cache-after-load` — weight-read performance tuning. - `--load-format runai_streamer` + `{"distributed", "concurrency", "memory_limit"}` — object storage streaming. - `--load-format dummy --wait-for-initial-weights` + `sglang.srt.checkpoint_engine.update` — checkpoint engine. - `--load-format remote_instance` + `--remote-instance-weight-loader-*` — R-Fork. ## When To Use - Multi-LoRA serving: many fine-tuned variants of one base model behind a single deployment (e.g. per-tenant or per-task adapters). - LoRA GPU pinning: a small set of adapters receiving disproportionate traffic. - LoRA overlap loading: only once profiling shows adapter weight loading (not prefill compute) is the bottleneck. - `sharded_state`/`layered` load formats: very large TP models where full-checkpoint reads or peak memory during load are the bottleneck. - `runai_streamer`/object storage: models living in S3/GCS/Azure where a full local download is undesirable. - Checkpoint engine: large models or multi-node deployments where disk bandwidth or startup time dominates. - R-Fork: elastic/fast-scaling deployments that need to boot a new replica from an already-warm instance instead of cold-reading from disk. ## Risks & Pitfalls - Omitting `--max-lora-rank`/`--lora-target-modules` at startup works fine for static `--lora-paths` deployments but constrains dynamically loaded adapters to match or be smaller than the inferred shape — a rank-mismatch failure mode that only appears later. - Pinning too many LoRA adapters can halt unpinned request scheduling in the extreme case; SGLang's `max-loras-per-batch - 1` cap prevents total starvation but doesn't prevent degraded performance from over-pinning. - LoRA overlap loading is off by default for good reason — enabling it under a workload where prefill compute dominates adapter load time actively **increases** TTFT (see the worked 28ms vs. 82ms example above) by breaking multi-adapter prefill batching. - `--weight-loader-prefetch-checkpoints` disables multithreaded safetensors loading by default (to avoid I/O oversubscription) — re-enabling `enable_multithread_load` on local NVMe (where prefetch is a no-op) can silently regress if the two features fight over I/O bandwidth on network storage instead. - `fastsafetensors`' GPUDirect Storage (`enable_gds`, default on) fails on hosts without the NVIDIA GDS kernel driver (e.g. gVisor sandboxes) — must be explicitly disabled there. - Object storage streaming (`runai_streamer`) only supports `.safetensors` — other weight formats are simply unsupported, with no fallback. Distributed streaming itself is limited to CUDA-like devices; on other devices it falls back to non-distributed streaming rather than failing. - Checkpoint Engine's documented launch pattern requires the SGLang server to start with **both** `--load-format dummy` (lets it proceed to CUDA graph capture and other init work with placeholder weights, enabling the overlap) **and** `--wait-for-initial-weights` (waits for the checkpoint engine to supply real weights before becoming ready) — omitting `--load-format dummy` forfeits the overlap benefit for that documented workflow. ## Related Concepts - [[concepts/quantization]] — `bitsandbytes` load format and QLoRA (`qlora_adapter_name_or_path`) combine quantized base weights with a LoRA adapter; `flash_rl` load format applies native FP8 quantization during load for RL training. - [[concepts/parallelism-and-disaggregation]] — LoRA serving composes with TP; checkpoint engine's `--inference-parallel-size` scales with aggregate TP across nodes; R-Fork is a boot-time alternative to cold-loading in elastic multi-instance deployments. - [[concepts/structured-outputs-and-tool-calling]] — `--chat-template` selection (mentioned there for tool/reasoning parsers) is a related per-model loading concern. ## Sources - raw/github_doc-docs-docs-advanced-features-lora-mdx.md - raw/github_doc-docs-docs-advanced-features-model-loading-mdx.md - raw/github_doc-docs-docs-advanced-features-object-storage-mdx.md - raw/github_doc-docs-docs-advanced-features-checkpoint-engine-mdx.md - raw/github_doc-docs-docs-advanced-features-rfork-mdx.md --- title: "Observability and Determinism" type: concept tags: [architecture, operator, advanced, well-established] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-observability-mdx.md" - "raw/github_doc-docs-docs-advanced-features-deterministic-inference-mdx.md" - "raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md" confidence: medium --- # Observability and Determinism ## Definition Three operational concerns for running SGLang in production: **observability** (Prometheus metrics, request logging, request/crash dump-and-replay for debugging), **deterministic inference** (making outputs bit-identical across runs despite dynamic batching, critical for RL training and testing), and **hyperparameter tuning** (reading the server's own log line to diagnose and fix throughput bottlenecks via memory, batching, and parallelism knobs). ## How It Works ### Observability **Metrics**: `--enable-metrics` exposes Prometheus metrics at `curl http://localhost:30000/metrics`. See the separate Production Metrics and Production Request Tracing references for the full metric catalog (out of scope for this page). **Logging**: request contents are **not** logged by default; enable with `--log-requests`, control verbosity with `--log-request-level`. Change verbosity live without restarting: `python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --log-level=debug`. **Request dump and replay**: capture live traffic for later benchmarking/debugging via the same `configure_logging` utility: `--dump-requests-folder /tmp/sglang_request_dump --dump-requests-threshold 100` dumps a pickle file every 100 requests. Replay with `scripts/playground/replay_request_dump.py`. **Crash dump and replay**: set `--crash-dump-folder /tmp/crash_dump` to preserve recent request data across a crash, plus (on NVIDIA CUDA) collect device coredumps for low-level debugging. On crash, SGLang writes completed requests retained in the crash-dump buffer plus in-flight requests, server arguments, and the launch command to `/tmp/crash_dump//crash_dump_.pkl` — replayable with the same `replay_request_dump.py` script. On CUDA, this option also sets default environment variables *before CUDA initializes* to enable device coredumps on CUDA exceptions and let SGLang trigger device coredumps for live scheduler processes when handling a crash; these dumps land at `/tmp/crash_dump//core.cuda..`. Explicitly set CUDA coredump env vars (including a custom `CUDA_COREDUMP_FILE`) take precedence over these defaults. This option does **not** configure OS-level process core dumps — only CUDA device coredumps. ### Deterministic inference Standard LLM inference is non-deterministic even at `temperature=0` because **varying batch sizes** cause GPU kernels to split reduction operations differently, changing floating-point addition order — and floating-point addition is non-associative (`(a+b)+c ≠ a+(b+c)`), so identical logical inputs can produce different numeric results depending on what else is in the batch. This matters for: **RL training** (consistent logprobs reduce stochastic noise, making training more stable/reproducible/debuggable), **testing/debugging** (reproducible validation), and **production** (reliability and consistent UX). SGLang's fix builds on Thinking Machines Lab's **batch-invariant operators**, achieving full determinism while remaining compatible with chunked prefill, CUDA graphs, radix cache, and non-greedy sampling (roadmap: sgl-project/sglang#10278). Supported only on three attention backends, with this compatibility matrix: | Backend | CUDA Graph | Chunked Prefill | Radix Cache | Non-greedy Sampling | |---|---|---|---|---| | FlashInfer | Yes | Yes | **No** | Yes | | FA3 | Yes | Yes | Yes | Yes | | Triton | Yes | Yes | Yes | Yes | Enable with `--enable-deterministic-inference`, using one of the three supported backends via `--attention-backend` (`flashinfer`, `fa3`, or `triton`); the documented default is `fa3`: ```bash python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B --attention-backend fa3 --enable-deterministic-inference ``` Works for dense (Llama, Qwen3-8B) and MoE models (Qwen3-30B-A3B) alike, always paired with an explicit compatible attention backend. **Determinism with non-greedy sampling (temperature > 0)**: when deterministic inference is enabled (`--enable-deterministic-inference`), omitting `sampling_seed` uses SGLang's default fixed sampling seed of `42`, so a given prompt+temperature combination reproduces the same output every run. This reproducibility guarantee does **not** apply unless deterministic inference is explicitly turned on — standard inference can vary even at temperature 0 (see above), so a default sampling seed alone does not make requests reproducible by default. For workloads needing *multiple distinct but reproducible* responses per prompt (e.g. GRPO — Group Relative Policy Optimization — in RL), pass an explicit `sampling_seed` per request: ```python for seed in [42, 43, 44, 45, 46]: requests.post("http://localhost:30000/generate", json={ "text": "Tell me a joke", "sampling_params": {"temperature": 0.8, "max_new_tokens": 128, "sampling_seed": seed}, }) ``` Different seeds → diverse responses; same seed → identical response across runs. **Verification**: `python3 -m sglang.test.test_deterministic` with `--test-mode {single,prefix,radix_cache}` (single = same prompt across varying batch sizes; prefix = prompts with different prefix lengths; radix_cache = cached vs. uncached prefill consistency). The documented invocations pass `--n-trials 50` for the `single` and `prefix` modes; the `radix_cache` mode's documented invocation omits `--n-trials`. Expected: `Unique samples: 1` for every test — anything else indicates the determinism guarantee is broken for that configuration. ### Hyperparameter tuning (offline batch throughput) The central lever for offline batch throughput is achieving a **large batch size**. Diagnose from the server's own steady-state log line: ``` Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317 ``` **`#queue-req`** (requests waiting): frequent `0` means the client is submitting too slowly — healthy range is **100–2000**; too large increases server-side scheduling overhead. Fix on the client side (submission rate), not the server. **`token usage`** (KV cache memory utilization): `>0.9` is good utilization. - Frequently `<0.9` with `#queue-req > 0` → server is too conservative about admitting new requests (common when clients request a large `max_new_tokens` but responses stop early via EOS/stop strings) → decrease `--schedule-conservativeness` (e.g. to `0.3`). - Very high usage plus frequent `KV cache pool is full. Retract requests. #retracted_reqs: ..., #new_token_ratio: ...` warnings → increase `--schedule-conservativeness` (e.g. to `1.3`). Occasional retraction (~once/minute) is fine. **`--mem-fraction-static`**: total GPU memory = model weights + KV cache pool + CUDA graph buffers + activations; `mem_fraction_static = (weights + KV cache pool) / GPU capacity`. Maximize it to grow the KV cache pool, while leaving enough for activations/CUDA-graph buffers — SGLang's default is heuristic, not necessarily optimal. Rule of thumb: 5–8 GB reserved for activations is typically enough; check the startup log's `available_gpu_mem` value (e.g. `max_total_num_tokens=665690, chunked_prefill_size=8192, ..., available_gpu_mem=13.50 GB`): 5–8 GB is good, 10–20 GB means raise `--mem-fraction-static` to give the KV pool more room, and too low risks OOM later so lower it. Alternatively, increment `--mem-fraction-static` by `0.01` steps until OOM appears, then back off. **OOM avoidance** — three levers, applied by symptom: OOM during **prefill** → lower `--chunked-prefill-size` (e.g. to `4096` or `2048`; trades prefill speed on long prompts for memory). OOM during **decode** → lower `--max-running-requests`. Either phase → lower `--mem-fraction-static` (e.g. `0.8` or `0.7`), which shrinks the KV pool and thus concurrency/peak throughput as a side effect. **`--cuda-graph-max-bs-decode`**: CUDA graph is enabled by default only for small batch sizes (roughly <160–256); some models — especially at large TP — benefit from CUDA graph up to batch sizes of 512–768, so raising this can help. CUDA graph buffers consume memory, so raising `--cuda-graph-max-bs-decode` often requires simultaneously lowering `--mem-fraction-static`. **`--dp-size` / `--tp-size`**: data parallelism favors throughput over tensor parallelism when GPU memory allows — prefer DP when there's enough memory to replicate. For DP beyond the basic `--dp-size` flag, use SGLang Model Gateway (SMG) for better production routing (see [[concepts/parallelism-and-disaggregation]]). **Other levers**: `--enable-torch-compile` accelerates small models at small batch sizes; quantization (e.g. `--quantization fp8`, see [[concepts/quantization]]) trades precision for throughput; other parallelism strategies (expert parallelism, or `--enable-dp-attention --dp-size 8` for DeepSeek-family models, see [[concepts/parallelism-and-disaggregation]]); `--schedule-policy lpm` (longest prefix match) reorders requests to increase cache hits on shared-prefix-heavy workloads, at the cost of extra scheduling overhead. ## Key Parameters - `--enable-metrics`, `--log-requests`, `--log-request-level` — basic observability toggles. - `--dump-requests-folder`, `--dump-requests-threshold`, `--crash-dump-folder` — dump/replay tooling (configured live via `sglang.srt.managers.configure_logging`). - `--enable-deterministic-inference`, `--attention-backend {flashinfer,fa3,triton}`, `sampling_seed` (per-request) — determinism. - `--schedule-conservativeness`, `--mem-fraction-static`, `--chunked-prefill-size`, `--max-running-requests`, `--cuda-graph-max-bs-decode`, `--dp-size`/`--tp-size`, `--schedule-policy lpm` — throughput tuning. ## When To Use - Enable `--enable-metrics` by default in any production deployment. `--log-requests` is off by default upstream (request contents are not logged unless enabled) — turning it on broadly is curator advice, not an upstream recommendation, and should come with a privacy/retention/security caveat since prompts can contain sensitive data; scope it to what the deployment's data-handling policy allows. Similarly, adding `--crash-dump-folder` wherever crash root-causing matters is curator advice, not an upstream universal requirement. - Use deterministic inference for RL training pipelines (stable logprobs), CI/regression testing (reproducible outputs), or any production case where identical inputs must yield identical outputs. - Use `sampling_seed` per-request specifically for RL algorithms (GRPO) needing multiple diverse-but-reproducible completions per prompt. - Apply the hyperparameter tuning guidance whenever throughput is below expectations in offline batch inference — start from the steady-state log line, not blind parameter sweeps. ## Risks & Pitfalls - Deterministic inference is only supported on FlashInfer, FA3, and Triton attention backends. The source does not document what happens if an unsupported backend is selected with `--enable-deterministic-inference` (accepted-but-ignored, rejected, or undefined behavior) — treat any other backend as simply unsupported for this feature rather than assuming a specific failure mode. - FlashInfer specifically does **not** support determinism together with radix cache — combining `--enable-deterministic-inference --attention-backend flashinfer` with prefix caching breaks the determinism guarantee that backend otherwise provides for CUDA graph/chunked prefill/non-greedy sampling. - The default sampling seed (`42`) only guarantees reproducibility when `--enable-deterministic-inference` is explicitly turned on — it does not make requests reproducible by default on a standard (non-deterministic-inference) deployment, where outputs can still vary at any temperature including 0. Teams relying on deterministic inference for reproducibility, but expecting natural sampling diversity per identical request, must explicitly vary `sampling_seed`. - Raising `--cuda-graph-max-bs-decode` without correspondingly lowering `--mem-fraction-static` can cause OOM, since CUDA graph buffers themselves consume more memory at larger captured batch sizes. - Lowering `--mem-fraction-static` to fix OOM directly shrinks KV cache capacity, trading away concurrency/peak throughput — it's a blunt instrument, not a free fix. - `--schedule-policy lpm` improves cache hit rate but adds scheduling overhead — only worth it when the workload genuinely has many shared prefixes. - CUDA crash-dump environment variables set implicitly by `--crash-dump-folder` are overridden by any explicitly-set CUDA coredump env var (including `CUDA_COREDUMP_FILE`) — a pre-existing env var can silently change where/whether coredumps land. ## Related Concepts - [[concepts/attention-backends-and-cuda-graph]] — deterministic inference's backend restriction (FlashInfer/FA3/Triton only) and CUDA graph batch-size tuning are direct extensions of that page's material. - [[concepts/hierarchical-caching]] — radix cache interacts with determinism (FlashInfer incompatibility) and with `--schedule-policy lpm`'s cache-hit-oriented reordering. - [[concepts/parallelism-and-disaggregation]] — DP vs. TP throughput tradeoff and SMG routing referenced in the tuning guidance; SMG also exposes Prometheus metrics relevant to this page's observability material. - [[concepts/quantization]] — FP8 quantization listed as a throughput lever alongside the memory/batching knobs here. ## Sources - raw/github_doc-docs-docs-advanced-features-observability-mdx.md - raw/github_doc-docs-docs-advanced-features-deterministic-inference-mdx.md - raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md --- title: "Offline Engine" type: concept tags: [basic-usage, api, foundational, user, developer] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-basic-usage-offline-engine-api-mdx.md", "raw/github_doc-docs-docs-basic-usage-native-api-mdx.md", "raw/github_doc-docs-docs-get-started-quickstart-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition The Offline Engine API is SGLang's direct, in-process inference interface — the `sglang.Engine` class — that runs batch inference without launching an HTTP server. It exists for cases where the HTTP server layer adds unnecessary complexity or overhead (raw/github_doc-docs-docs-basic-usage-offline-engine-api-mdx.md). Its two general use cases are offline batch inference, and building a custom server on top of the engine. ## How It Works Instantiate the engine directly with a model path: ```python import sglang as sgl llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct") ``` The engine supports four inference modes, all documented with worked examples: ### Non-streaming synchronous generation ```python prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### Streaming synchronous generation Uses the `stream_and_merge` helper from `sglang.utils` to merge overlapping streamed chunks per prompt: ```python from sglang.utils import stream_and_merge for prompt in prompts: merged_output = stream_and_merge(llm, prompt, sampling_params) print("Generated text:", merged_output) ``` ### Non-streaming asynchronous generation ```python import asyncio async def main(): outputs = await llm.async_generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"Prompt: {prompt}\nGenerated text: {output['text']}") asyncio.run(main()) ``` ### Streaming asynchronous generation Uses `async_stream_and_merge` to iterate cleaned (non-overlapping) chunks as they arrive: ```python from sglang.utils import async_stream_and_merge async def main(): for prompt in prompts: async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params): print(cleaned_chunk, end="", flush=True) asyncio.run(main()) ``` ### Shutdown ```python llm.shutdown() ``` ### Nested event loops Using the engine inside IPython or other code that already runs a nested asyncio loop requires patching the loop first: ```python import nest_asyncio nest_asyncio.apply() ``` ### Advanced usage The engine also supports VLM (vision-language model) inference and extracting hidden states, both demonstrated in dedicated example scripts in the SGLang source tree (`examples/runtime/engine/offline_batch_inference_vlm.py` and `examples/runtime/hidden_states/`), referenced but not reproduced in the raw source for this page. A full custom-server example built on top of the engine lives at `examples/runtime/engine/custom_server.py`. ### Relationship to the quickstart flow The get-started quickstart frames offline batch inference as an explicit alternative to the server flow, using the identical `Engine` pattern shown above with `llm.generate(prompts, sampling_params)` followed by `llm.shutdown()` (raw/github_doc-docs-docs-get-started-quickstart-mdx.md). ### Relationship to native server endpoints The native API's `/generate` endpoint (see [[concepts/server-apis]]) is documented as similar to OpenAI's `/v1/completions` — the source does not describe it as the HTTP-served equivalent of `Engine.generate`, only as a comparable text-generation operation over a different transport. Native API endpoints like `/encode` (embeddings) and `/classify` (reward models) are documented as server routes requiring a running server; these offline-engine docs simply don't document a Python-object equivalent for them (raw/github_doc-docs-docs-basic-usage-native-api-mdx.md). ## Key Parameters - `model_path` — the constructor keyword argument used in every documented example, e.g. `sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")`. - `sampling_params` — a dict (e.g. `{"temperature": 0.8, "top_p": 0.95}`) passed to `generate`/`async_generate` in the documented examples; see [[concepts/sampling-parameters]] for the full parameter reference. - `prompts` — a list of prompt strings, as shown in the documented examples; a single-string form is not demonstrated in these sources. ## When To Use - Batch offline inference over a fixed set of prompts, where no concurrent HTTP clients are needed. - Embedding SGLang inference directly inside another Python process or pipeline (e.g., a data-processing job) without the overhead of a network hop. (An RL training loop is a plausible use of the same pattern, but is an illustrative inference, not a use case documented in these sources.) - Building a custom server with bespoke routing/auth/business logic on top of SGLang's engine, rather than using the built-in HTTP server. - VLM batch inference or hidden-state extraction workflows that use the example scripts referenced above. ## Risks & Pitfalls - Running the engine inside IPython, Jupyter, or other environments with an existing nested event loop needs `nest_asyncio.apply()` called first, per the documented usage note (the source frames this as a requirement, not a specific failure mode). - The offline engine has no HTTP surface, so operational endpoints documented for a running server — cache flush, weight hot-swap from disk, health checks, expert-distribution recording, tokenize/detokenize (see [[concepts/server-apis]]) — are not documented as available through this API; whether or how each one maps onto the Engine object is not addressed in these sources. - The documented examples all call `llm.shutdown()` at the end of the script; the sources don't state what happens if it's omitted, so follow the examples and call `shutdown()` when finished. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/sending-requests]] - [[concepts/server-apis]] - [[concepts/sampling-parameters]] - [[concepts/installation]] ## Sources - raw/github_doc-docs-docs-basic-usage-offline-engine-api-mdx.md - raw/github_doc-docs-docs-basic-usage-native-api-mdx.md - raw/github_doc-docs-docs-get-started-quickstart-mdx.md --- title: "Parallelism and Disaggregation" type: concept tags: [parallelism, architecture, advanced, operator, well-established] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-expert-parallelism-mdx.md" - "raw/github_doc-docs-docs-advanced-features-pipeline-parallelism-mdx.md" - "raw/github_doc-docs-docs-advanced-features-dcp-mdx.md" - "raw/github_doc-docs-docs-advanced-features-dp-dpa-smg-guide-mdx.md" - "raw/github_doc-docs-docs-advanced-features-dp-for-multi-modal-encoder-mdx.md" - "raw/github_doc-docs-docs-advanced-features-pd-disaggregation-mdx.md" - "raw/github_doc-docs-docs-advanced-features-epd-disaggregation-mdx.md" confidence: medium --- # Parallelism and Disaggregation ## Definition SGLang scales a single logical model across many GPUs (and many *stages* of the request lifecycle) using several composable parallelism strategies — Tensor Parallelism (TP), Pipeline Parallelism (PP), Expert Parallelism (EP), Data Parallelism (DP), Data Parallelism Attention (DPA), and Decode Context Parallelism (DCP) — plus two disaggregation architectures that split the request lifecycle across specialized instance pools: Prefill–Decode (PD) disaggregation and Encoder–Prefill–Decode (EPD) disaggregation for vision-language models. These strategies are designed to be composed (e.g., TP+DPA+EP+DCP together for a single DeepSeek deployment) rather than chosen exclusively. ## How It Works ### Tensor Parallelism (TP) and Data Parallelism (DP) — the baseline - **TP** shards model weights (attention heads, MLP columns) across GPUs; all GPUs in a TP group cooperate on every request via collectives (all-reduce/all-gather). - **DP** replicates the full model across GPU sets, routing different requests/batches to each replica independently. Set with `--dp-size N`. - **Native DP** (`python -m sglang.launch_server --dp-size N`) uses an in-process `DataParallelController` with only basic load-balancing (`round_robin`, `total_requests`, `total_tokens`), no cache-awareness, and is explicitly **not recommended for production** (only used by some legacy RL frameworks). - **SMG-based DP** (recommended): launch with `python -m sglang_router.launch_server --dp-size N` (or a separate `sglang_router.launch_router --worker-urls ... --policy cache_aware`). SGLang Model Gateway (SMG, formerly "SGLang DP Router," built in Rust) adds cache-aware routing (approximate radix tree per worker, routes to highest prefix match, falls back to shortest-queue), circuit breakers/health checks, 40+ Prometheus metrics, hot worker add/remove, and multi-node support. Benchmarked at +92% throughput and cache hit rate rising from 20%→75% vs. native DP on a shared-prefix workload — a historical SGLang v0.4 blog benchmark (8×A100 80GB, dp-size=8, multiple long-prefix groups), not a current-release measurement. Load-balancing policies: `cache_aware` (default/recommended), `round_robin`, `random`, `power_of_two`. ### Data Parallelism Attention (DPA) DPA applies data parallelism specifically to the attention component rather than the whole model. It matters most for **Multi-Head Latent Attention (MLA)** models (DeepSeek family, MiniMax, Kimi-K2) which have only one KV head — under plain TP that KV cache gets duplicated across every TP rank, wasting memory and capping batch size. DPA gives every DP replica its own independent KV cache (no duplication), each replica can be in a different forward mode (prefill/decode/idle) simultaneously, and it also works for standard-attention models (e.g., Qwen). Enable with **both** `--dp-size N` and `--enable-dp-attention`; `--dp-size` must be `>1` (DPA silently disables if `dp_size==1`) and `tp_size % dp_size == 0` must hold: ```bash python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 --dp-size 8 --enable-dp-attention ``` DPA is commonly paired with EP for MoE models but does not require it. A related optimization, **TP LM-Head all-to-all** (`--enable-tp-lm-head-all-to-all`, default-on for decode nodes and pure DPA; opt out with `--no-enable-tp-lm-head-all-to-all`), replaces the heavy all-gather over the full vocabulary in `tp-lm-head` with a lighter all-to-all that only exchanges local sharded-vocab rows. ### Expert Parallelism (EP) for MoE EP distributes MoE expert weights across GPUs. Two independent flags control it: - `--moe-a2a-backend` — all-to-all communication backend: `none` (default; All-Reduce/All-Gather, for hybrid EP+TP), `deepep` (DeepEP, large-scale EP), `mooncake` (DeepEP extension for elastic/RDMA EP), `nixl` (NIXL-EP, elastic + fault tolerant), `mori` (AMD ROCm native), `flashinfer`, `ascend_fuseep` (Ascend NPU), `pplx` (Perplexity's NVSHMEM kernels, low-latency/masked only, FP8+Hopper, requires `--enable-dp-attention` with ≥2 DP groups). - `--moe-runner-backend` — MoE computation backend: `auto` (default, picks by hardware/quantization), `triton`, `deep_gemm` (FP8 block-wise, large-scale EP), `cutlass`, `flashinfer_trtllm`, `flashinfer_trtllm_routed`, `flashinfer_cutlass`, `flashinfer_mxfp4`, `flashinfer_cutedsl`. DeepEP/Mooncake support `normal` mode (throughput-optimized, prefill) and `low_latency` mode (CUDA-Graph compatible, decode); `--deepep-mode auto` switches dynamically at runtime. `deepep`, `mooncake`, `nixl`, `ascend_fuseep`, `pplx`, and `mori` all require `ep_size == tp_size`; hybrid EP+TP (`ep_size < tp_size`) only works with the `none` backend. ```bash python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 \ --moe-a2a-backend deepep --moe-runner-backend deep_gemm --tp 8 --ep 8 ``` The EP framework decouples the MoE forward pass into dispatch → pre-permute → core runner → post-permute → combine around a `FusedMoE` entry point (`BaseDispatcher`, `MoeRunnerCore`, `PermuteMethodPool`, backend-agnostic TopK router), so new backends can be added without touching core logic. **Overlap techniques**: Two-Batch Overlap (`--enable-two-batch-overlap`) splits requests into micro-batches and interleaves attention compute with dispatch/combine communication for up to 2x throughput. Single-Batch Overlap (`--enable-single-batch-overlap`) uses a dispatcher-hook system to overlap operations (e.g., shared-expert compute with DeepEP combine) within a single batch. **Load balancing**: `--enable-eplb` turns on the Expert Parallelism Load Balancer (EPLB, from DeepSeek), which analyzes expert activation stats and rearranges/replicates experts to minimize GPU-utilization variance; pair with larger batch sizes and periodic rebalancing (e.g., every 1000 requests). Ascend NPU notes: `--moe-a2a-backend` supports only `deepep` and `ascend_fuseep` (the latter fuses dispatch↔combine ops and is decode-only in PD mode); `--moe-runner-backend` does not need to be configured (auto-handled on Ascend, not an unconfigurable parameter); DeepEP Ascend adds an "ant-moving" streaming-batch transmission mode for long sequences via `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`, `DEEPEP_NORMAL_LONG_SEQ_ROUND`, `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`, tuned against the `HCCL_BUFFSIZE` buffer-size formula. ### Data Parallelism for the multimodal (vision) encoder VLMs pair a small Vision Transformer (ViT) encoder with a large LLM decoder. Because the ViT is small, TP on it yields little benefit but still pays the all-reduce cost every layer. SGLang instead runs the ViT in **data parallel** while the LLM stays tensor-parallel — lowering TTFT and raising throughput. Enable with `--mm-enable-dp-encoder`: ```bash python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --tp 2 --mm-enable-dp-encoder ``` Supported models: Qwen2.5-VL, Qwen3-VL, InternVL, GLM-4.5V/4.6V. ### Pipeline Parallelism (PP) for long context TP handles intra-node scaling but hits communication bottlenecks across nodes; PP only needs cross-node communication at stage boundaries, so it composes better with multi-node long-context serving and reduces Time to First Token (TTFT) for very long inputs. `--pp-size N` splits model layers into stages. SGLang's PP implementation uses a **Micro-batching Event Loop** with non-blocking async peer-to-peer transfer (`async_send` returns a `P2PWork` handle; sync is deferred to `_pp_commit_comm_work`) plus dedicated `forward_stream` and `copy_stream` (alongside `default_stream`) so GPU compute, D2H copies, and inter-stage communication overlap. PP with Dynamic Chunked Prefill "has the potential" to reduce TTFT for long-context inputs, per the docs' own framing — PP is called a "promising" parallelization strategy, not a guaranteed win, and its benefit depends on workload and hardware (e.g., TP still wins when cross-node communication isn't the bottleneck). **Dynamic chunked prefill** (`--enable-dynamic-chunking`) addresses pipeline bubbles from non-uniform per-chunk runtime (attention cost scales with prefix length `L`): it predicts the next chunk size so `Runtime(L + next) - Runtime(L) = Runtime(initial)`, rounding down to a multiple of `max(--page-size, 64)`. `--chunked-prefill-size` sets the *initial* chunk size (should be 2–3× the optimal fixed chunk size when dynamic chunking is on). `SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR` (default 0.75, recommended range 0.6–0.85) controls how aggressively chunk size follows the quadratic prediction model (`1` = follow strictly, `0` = fixed-size chunking). Tip: put the larger layer partition on the higher PP rank when layers don't divide evenly (e.g. `SGLANG_PP_LAYER_PARTITION=15,15,15,16` beats `16,15,15,15` for DeepSeek-V3.1) to reduce bubbles. The docs explicitly flag dynamic chunking as **experimental**: it "requires a certain amount of tuning experimentation and may not be suitable for all workloads." Treat the tuning guidance above as a starting point, not a guaranteed configuration. Example (DeepSeek-V3.1, 128K input, 4 nodes, TP8×PP4): ```bash python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 --attention-backend fa3 --host 0.0.0.0 \ --watchdog-timeout 3600 --max-running-requests 128 --chunked-prefill-size 12288 \ --enable-dynamic-chunking # with export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.65 ``` (`--disable-radix-cache` here is only for reproducible benchmarking, not recommended in production.) PP + PD disaggregation best practices are not yet documented upstream ("to be added"). ### Decode Context Parallelism (DCP) DCP stripes a request's **MLA KV cache** by token position across ranks within an existing TP group, complementing (not replacing) TP and DPA: with DCP size `c`, rank `r` owns position `p` where `p mod c = r`, so each rank stores/reads roughly `1/c` of the cache — growing usable long-context KV capacity per GPU. Each rank's local attention produces a partial output and log-sum-exp (LSE); these are exchanged in one packed all-to-all and merged exactly aside from floating-point reduction order: `lse = logsumexp_r(lse_r)`, `o = Σ_r exp(lse_r - lse) · o_r`. Key flags: | Setting | Behavior | |---|---| | `--dcp-size N` (alias `--decode-context-parallel-size`) | Enables DCP; widens virtual capacity and page size by `N` | | `--dcp-comm-backend` | `ag_rs` (generic fallback: query AG + LSE AG + FP32 output RS), `a2a` (packed NCCL A2A; with replicated Q, one collective — the canonical low-latency path), `fi_a2a` (FlashInfer MNNVL A2A, needs CUDA/SM90+/MNNVL fabric) | | `--dcp-replicate-q-proj` / `--no-dcp-replicate-q-proj` | Gathers query-projection and `w_kc` weights once at startup to skip the per-layer query all-gather (BF16/FP16 only) | Topology constraint: a DCP group must fit inside one attention-TP group and one attention-DP replica — `attn_tp_size = tp_size / attn_dp_size` and `attn_tp_size % dcp_size == 0` (e.g. TP=64/DP=4/DCP=16 is valid; DCP=32 is not). **Startup only checks `tp_size % dcp_size == 0`**, not the full condition — DPA+DCP deployments must self-enforce containment. ```bash sglang serve --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code --tp-size 8 --dcp-size 8 --host 0.0.0.0 --port 30000 ``` Kimi K3 example (DPA + DCP + EP): `--tp-size 32 --ep-size 32 --enable-dp-attention --dp-size 4 --dcp-size 8` — the model override auto-enables replicated-Q and picks `fi_a2a` on MNNVL systems (else `a2a`), decode backend `cutedsl_mla`. **Compositions**: DCP × speculative decoding replicates the draft KV cache in full on every rank (DCP only stripes target MLA KV, not draft); target verify/decode use `cutedsl_mla`. DCP × PD disaggregation requires token relayout when prefill DCP=1 transfers into a striped decode pool (equal DCP sizes reuse the existing page path; other size transitions are rejected); it additionally requires Mooncake or NIXL, matching physical page size and KV dtype, prefill attention CP=1, and decode chunk cache — decode radix cache and HiCache are not supported in this combination. DCP × HiCache L2 keeps one widened logical page space across L1/L2 (see [[concepts/hierarchical-caching]]); L3, LMCache, HiSparse, non-MLA host pools, speculative decoding, and PD decode are **not** supported in this combination. On Kimi K3, DCP applies only to MLA layers — request-indexed KDA (linear-attention) state is unaffected and does not gain concurrency headroom from DCP. ### Prefill–Decode (PD) Disaggregation LLM inference has two phases with opposite resource profiles: **Prefill** (compute-bound, processes the whole input) and **Decode** (memory-bound, KV-cache-heavy, one token at a time). Running them in one unified engine causes prefill batches to interrupt in-flight decode batches, and can imbalance DP-attention workers (one doing prefill while its peer decodes). PD disaggregation runs prefill and decode as **separate instance pools** connected by a KV-cache transfer engine, each independently tunable and scalable. Two transfer engines are supported: **Mooncake** and **NIXL** (plus an Ascend-specific backend). Minimal single-node example: ```bash python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill --port 30000 --disaggregation-ib-device mlx5_roce0 python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode --port 30001 --base-gpu-id 1 --disaggregation-ib-device mlx5_roce0 python -m sglang_router.launch_router --pd-disaggregation \ --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 ``` Multi-node DeepSeek deployments add `--tp-size 16 --dp-size 8 --enable-dp-attention --moe-a2a-backend deepep` on both sides. NIXL is selected with `--disaggregation-transfer-backend nixl` (default NIXL backend is UCX; override via `SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC` or another installed plugin). Mooncake install: `uv pip install mooncake-transfer-engine`. `--disaggregation-ib-device` accepts a shared device list (`mlx5_0,mlx5_1`), a per-GPU JSON map (`{"0": "mlx5_0,mlx5_1", ...}`), or a path to such a JSON file. NVLink transport for KV transfers (recommended on NVL72) is enabled via `SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK` + `MC_FORCE_MNNVL=True`; intra-node NVLink (A100/H20/H100) via `SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK` + `MC_INTRANODE_NVLINK=true`. Auxiliary data still moves over TCP in both cases. Tuning environment variables: | Variable | Side | Default | Meaning | |---|---|---|---| | `SGLANG_DISAGGREGATION_THREAD_POOL_SIZE` | Prefill | `int(0.75*cpu_count())//8`, limited to be larger than 4 and less than 12 | Worker threads per TP rank for KV transfer | | `SGLANG_DISAGGREGATION_QUEUE_SIZE` | Prefill | `4` | Parallel transfer queues (decode requests sharded across them; `1` = strict FCFS) | | `SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT` | Prefill | `300`s | Timeout receiving destination KV indices at request init | | `SGLANG_DISAGGREGATION_BOOTSTRAP_ENTRY_CLEANUP_INTERVAL` | Prefill | `120`s | Cleanup interval for bootstrap entries | | `SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL` | Decode | `5.0`s | Health-check interval to prefill bootstrap servers | | `SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE` | Decode | `2` | Consecutive heartbeat failures before marking prefill offline | | `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` | Decode | `300`s | Timeout waiting for KV cache after request init | **Heterogeneous TP with GPU staging buffer**: when prefill and decode use different TP sizes (e.g. prefill TP=4, decode DP-attention TP=1), KV layouts differ between sides. The GPU staging buffer gathers KV head slices into a contiguous buffer on the prefill side, does one bulk RDMA transfer, then scatters into decode's KV pages — 2–5x throughput vs. per-token slicing at high concurrency, within ~5% of homogeneous-TP baselines. Enable the staging buffer specifically when prefill and decode use different TP sizes **with the Mooncake transfer backend** — set `SGLANG_DISAGG_STAGING_BUFFER=1` on **both** sides and size the decode-side ring buffer with `SGLANG_DISAGG_STAGING_POOL_SIZE_MB` (default `4096`). It auto-bypasses when TP sizes already match (even if enabled), and is **not** for MLA models (GQA/MHA only). Ascend backend uses `memfabric_hybrid` with `ASCEND_MF_STORE_URL`, or Mooncake via `ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE=true`; requires `ASCEND_NPU_PHY_ID` in the container env. For routing PD pools at scale, see the SGLang Model Gateway (router) integration referenced from this doc set (out of scope for this page — see the planned `server-arguments`/routing pages). Profiling prefill/decode workers separately is required due to torch profiler limitations (see the developer/benchmarking docs). ### Encoder–Prefill–Decode (EPD) Disaggregation VLM inference decomposes into three stages with different resource profiles: **Encoder** (compute-heavy ViT image encoding, needed only at request start), **Prefill** (initializes the LLM's KV cache from the full multimodal sequence), and **Decode** (memory-bandwidth-bound autoregressive generation). EPD separates all three into independently scalable instance pools, extending PD disaggregation with a dedicated encoder tier. Launch a language-only instance with `--language-only` (must also pass `--encoder-urls [ ...]`) or an encoder-only instance with `--encoder-only`. Encoder-transfer backends (`--encoder-transfer-backend`): `zmq_to_scheduler` (default), `zmq_to_tokenizer`, `mooncake`. ```bash # encoder python -m sglang.launch_server --model-path Qwen/Qwen3-VL-8B-Instruct --encoder-only \ --encoder-transfer-backend mooncake --port 30000 # language-only python -m sglang.launch_server --model-path Qwen/Qwen3-VL-8B-Instruct --language-only \ --encoder-urls http://127.0.0.1:30000 --encoder-transfer-backend mooncake --port 30002 ``` A **global multimodal embedding cache** (Mooncake-backed, `--enable-mm-global-cache` on the encoder server) lets repeated image inputs reuse previously computed ViT embeddings across instances instead of re-encoding — useful when images repeat across requests, encoder compute is the bottleneck, and Mooncake is already deployed. This is separate from `--encoder-transfer-backend` (which only governs output transport) and separate from the language-model KV cache / [[concepts/hierarchical-caching]]. Requires the same Mooncake env vars as elsewhere (`MOONCAKE_TE_META_DATA_SERVER`, `MOONCAKE_MASTER`, `MOONCAKE_PROTOCOL`, `MOONCAKE_GLOBAL_SEGMENT_SIZE`). Full three-tier example adds `--disaggregation-mode prefill` to the language-only server and a separate decode server, fronted by `sglang_router.launch_router --pd-disaggregation`. A gRPC encoder mode (`--grpc-mode` on the encoder, `SGLANG_ENCODER_MM_RECEIVER_MODE=grpc` on the prefill process) lets the encoder run as a gRPC server while prefill/decode stay HTTP. ## Key Parameters - `--tp`, `--dp-size`, `--enable-dp-attention`, `--ep`/`--ep-size`, `--pp-size`, `--dcp-size` — the core parallelism-degree flags; must satisfy divisibility constraints (`tp_size % dp_size == 0`, `attn_tp_size % dcp_size == 0`, `ep_size == tp_size` for most EP backends). - `--moe-a2a-backend`, `--moe-runner-backend`, `--deepep-mode` — EP communication/compute backend selection. - `--enable-two-batch-overlap`, `--enable-single-batch-overlap`, `--enable-eplb` — EP throughput/balance optimizations. - `--mm-enable-dp-encoder` — DP for the VLM vision encoder. - `--enable-dynamic-chunking`, `--chunked-prefill-size`, `SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR`, `SGLANG_PP_LAYER_PARTITION` — PP long-context tuning. - `--dcp-comm-backend`, `--dcp-replicate-q-proj` — DCP communication tuning. - `--disaggregation-mode {prefill,decode}`, `--disaggregation-transfer-backend {mooncake,nixl,ascend}`, `--disaggregation-ib-device` — PD disaggregation wiring. - `--encoder-only`, `--language-only`, `--encoder-urls`, `--encoder-transfer-backend`, `--enable-mm-global-cache` — EPD disaggregation wiring. ## When To Use - **DPA**: MLA models (DeepSeek, MiniMax, Kimi-K2) almost always; standard-attention models (Qwen) when KV memory pressure limits batch size. - **EP**: any MoE model too large to replicate whole experts per GPU, or where all-to-all expert routing beats replication. - **PP**: ultra-long-context, multi-node deployments where TP's per-layer collectives become the bottleneck (TTFT-sensitive) — the docs frame this as a "promising" strategy with "potential" gains, not a guaranteed win; outcome is workload- and hardware-dependent, and dynamic chunking specifically is experimental. - **DCP**: long-context MLA decode where KV cache capacity (not compute) is the limiting factor. - **PD disaggregation**: production serving at scale where prefill interruption of decode (or DP-attention imbalance) is hurting P50/P99 latency; lets prefill and decode scale/tune independently. - **EPD disaggregation**: image-heavy VLM workloads where vision encoding is the bottleneck and needs independent scaling from language prefill/decode. - **SMG-based DP** over native DP: essentially always in production — cache-aware routing, reliability, and observability all favor it. ## Risks & Pitfalls - DCP's startup check is weaker than the real topology constraint (`tp_size % dcp_size == 0` only) — misconfigured DPA+DCP topologies can silently cross replica boundaries. - EP backends other than `none` require `ep_size == tp_size`; hybrid EP+TP (`ep_size < tp_size`) is only supported with the `none` backend — the docs don't state what happens if you try another backend outside that constraint, so don't assume a specific fallback or error behavior; treat other configurations as unsupported. - GPU staging buffer for heterogeneous-TP PD transfer must **not** be enabled for MLA models (GQA/MHA only), is scoped to the Mooncake transfer backend, and auto-bypasses when TP sizes already match. - `--disable-radix-cache` shown in PP long-context benchmarks is for reproducibility only — do not carry it into production, since it forfeits prefix caching. - Dynamic chunked prefill (`--enable-dynamic-chunking`) is documented as an experimental feature requiring tuning and may not suit every workload — don't treat the tuning numbers above as universal defaults. - Native (naive) DP is explicitly called out as "highly not recommended for use right now" and only relevant to legacy RL frameworks — defaulting to it in new deployments loses cache-aware routing and reliability features. - Raising `SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT` (prefill-side) to tolerate slow KV transfer also delays how long a prefill instance takes to clean up memory held by a disconnected decode node. `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` (decode-side) only affects how long decode waits for KV cache after request init — the docs don't state a cleanup-delay consequence for it. - DCP × HiCache L2 / DCP × speculative decoding / DCP × PD each have narrow supported combinations (see composition notes above) — combinations like L3, LMCache, HiSparse, and non-MLA host pools with DCP are documented as **not supported**, not as producing a specific runtime error. ## Related Concepts - [[concepts/hierarchical-caching]] — HiCache L1/L2/L3 tiers, session radix cache, and HiSparse interact with DCP and with the KV transfer used by PD/EPD disaggregation. - [[concepts/attention-backends-and-cuda-graph]] — attention backend choice (e.g. `flashmla_sparse`, `cutedsl_mla`, `fa3`) determines which parallelism compositions (DCP, HiSparse) are available. - [[concepts/quantization]] — MoE runner backends (`deep_gemm`, `flashinfer_cutlass`, `flashinfer_mxfp4`) are quantization-scheme-specific. - [[concepts/speculative-decoding]] — composition constraints with DCP and with PD disaggregation (draft KV replication). - [[concepts/observability-and-determinism]] — SMG exposes 40+ Prometheus metrics and OpenTelemetry for monitoring parallel/disaggregated deployments. ## Sources - raw/github_doc-docs-docs-advanced-features-expert-parallelism-mdx.md - raw/github_doc-docs-docs-advanced-features-pipeline-parallelism-mdx.md - raw/github_doc-docs-docs-advanced-features-dcp-mdx.md - raw/github_doc-docs-docs-advanced-features-dp-dpa-smg-guide-mdx.md - raw/github_doc-docs-docs-advanced-features-dp-for-multi-modal-encoder-mdx.md - raw/github_doc-docs-docs-advanced-features-pd-disaggregation-mdx.md - raw/github_doc-docs-docs-advanced-features-epd-disaggregation-mdx.md --- title: "Quantization" type: concept tags: [quantization, advanced, well-established, operator] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-quantization-mdx.md" - "raw/github_doc-docs-docs-advanced-features-reasoning-aware-compression-mdx.md" - "raw/github_doc-docs-docs-advanced-features-quantized-kv-cache-mdx.md" - "raw/github_doc-docs-docs-advanced-features-expert-parallelism-mdx.md" - "raw/github_doc-docs-docs-advanced-features-speculative-decoding-mdx.md" confidence: medium --- # Quantization ## Definition SGLang reduces model-weight (and activation) precision to shrink memory footprint and raise throughput, via two broad approaches: **offline quantization** (loading a checkpoint already quantized by an external tool) and **online quantization** (SGLang quantizes BF16/FP16 weights itself at model load / dynamically at runtime). This is a separate axis from KV-cache quantization (see [[concepts/hierarchical-caching]]). A related but distinct technique, **Reasoning-Aware Compression (RAC)**, is a *pruning* (sparsity) calibration method specifically for reasoning models, using SGLang to generate the on-policy chain-of-thought calibration data. ## How It Works ### Offline vs. online quantization **Offline** quantization loads pre-quantized weights directly — required for methods like GPTQ/AWQ that pre-compute calibration statistics. **Online** quantization computes scaling parameters (e.g. weight max/min) dynamically at runtime, similar to FP8 delayed scaling. **Offline is recommended** for better performance, usability, and convenience. The **default** for a pre-quantized model is to omit `--quantization` — the quantization method is auto-parsed from the HF/msModelSlim config (e.g. DeepSeek V3/R1 ship already in FP8; adding `--quantization` there is redundant). This default has named exceptions, not a universal "always omit" rule: (1) for per-channel-quantized (INT8/FP8) models with per-token dynamic activation quantization, explicitly passing `--quantization w8a8_int8` or `--quantization w8a8_fp8` forces SGLang's own CUTLASS kernel (`W8A8Fp8Config`) instead of deferring to the checkpoint's declared quant config (e.g. vLLM's `CompressedTensorsConfig`); (2) for NVIDIA ModelOpt pre-quantized checkpoints, the docs instruct explicitly passing `--quantization modelopt_fp8` or `--quantization modelopt_fp4` (see the ModelOpt toolchain below) rather than relying on auto-detection. ### Platform / method support matrix | Method | NVIDIA | AMD (MI300X/325X/350X) | Ascend NPU (A2/A3/A5) | Notes | |---|---|---|---|---| | `fp8` | Yes | Yes | WIP | Aiter or Triton on AMD | | `mxfp4` | Yes | Yes | Yes (A5) | GPU needs CDNA3/CDNA4 (Aiter); Ascend A5 W4A4 MXFP4 for Qwen3 dense/MoE | | `mxfp8` | No | No | Yes (A5) | Ascend-only; Diffusion + LLM dense/MoE via CANN kernels | | `mxfp_w4a8` | No | No | Yes (A5) | Ascend-only W4A8 for Qwen3 dense | | `blockwise_int8` | Yes | Yes | No | Triton-based | | `w8a8_int8` / `w8a8_fp8` | Yes | Yes | No | Aiter/Triton FP8 on AMD | | `awq` / `gptq` | Yes | Yes | Yes | Triton/vLLM kernels on AMD, CANN on Ascend | | `compressed-tensors` | Yes | Yes | Partial | Ascend: FP8 not yet supported | | `quark` | Yes | Yes | No | AMD Quark, Aiter GEMM | | `auto-round` | Yes | Yes | Partial | Platform-agnostic (Intel) | | `quark_int4fp8_moe` | No | Yes | No | AMD-only online INT4→FP8 MoE (CDNA3/CDNA4) | | `awq_marlin` / `gptq_marlin` | Yes | No | No | Marlin kernels are CUDA-only | | `gguf` | Yes | No | Yes | sgl-kernel CUDA; Ascend does CPU pre-dequant | | `modelopt` / `modelopt_fp8` | Yes (Hopper/SM90+) | No | No | NVIDIA ModelOpt | | `modelopt_fp4` | Yes (SM80-90 via Marlin; SM100+ native) | No | No | NVIDIA ModelOpt | | `nvfp4_online` | Yes (Blackwell/SM100/SM103) | No | No | Online MoE-only NVFP4, per-token FP32 activation scales | | `petit_nvfp4` | No | Yes (MI250/300X/325X) | No | NVFP4 on ROCm via Petit; auto-selected loading NVFP4 on AMD | | `bitsandbytes` | Yes | Experimental | No | | | `modelslim` | No | No | Yes | Ascend-only, CANN kernels | On AMD, set `SGLANG_USE_AITER=1` where noted for Aiter acceleration. ### GEMM backend selection for FP4/FP8 Applies to blockwise FP8, MXFP8 (dense linear), and NVFP4 GEMM. `--fp8-gemm-backend`: `auto` (hardware-based auto-select), `deep_gemm` (SM90/SM100, JIT), `flashinfer_trtllm` (SM100, low-latency), `flashinfer_cutlass` (SM100/120), `flashinfer_deepgemm` (SM90, swapAB for small M/decoding), `cutlass` (SM120, sgl-kernel), `triton` (universal fallback), `aiter` (ROCm). `auto` order: DeepGEMM → FlashInfer TRTLLM → CUTLASS → AITER → Triton. `--fp4-gemm-backend`: `auto` (SM80+: `flashinfer_cutedsl` on SM100, `marlin` on SM80-90, `flashinfer_cutlass` otherwise), `flashinfer_cutlass`, `flashinfer_cudnn` (needs CUDA 13+/cuDNN 9.15+), `flashinfer_cutedsl` (SM100), `flashinfer_trtllm` (SM100), `marlin` (SM80-90 weight-only W4A16 fallback for NVFP4 checkpoints). NVFP4 GEMM requires FlashInfer installed. ### Offline quantization toolchains ```bash python3 -m sglang.launch_server --model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 --port 30000 --host 0.0.0.0 ``` - **Unsloth** — recommended tool for quantizing + loading. - **auto-round** (`pip install auto-round`) — schemes `W2A16`…`W8A16`, `NVFP4`, `MXFP4` (no real kernels), `GGUF:Q4_K_M`, etc.; also usable for VLMs (`AutoRoundMLLM`). CLI and SGLang-API paths exist. Known issues: mixed-bit quantization unsupported (vLLM layer fusion conflicts), limited quantized-MoE support (skip `mlp.gate` layer quantization if it errors), some quantized-VLM format combos fail (e.g. Qwen2.5-VL-7B `auto_round:auto_gptq` accuracy collapses to ~0; GPTQ format errors on output-size mismatch; `auto_round:auto_awq`/AWQ format work). SGLang-API path only supports `auto-round-int8` currently. CPU serving of 4-bit AutoRound checkpoints on Intel AMX: `SGLANG_USE_CPU_ENGINE=1 --quantization auto-round --device cpu`. - **GPTQModel** (`pip install gptqmodel --no-build-isolation`) — calibration-dataset-driven GPTQ quantization script. - **LLM Compressor** (`pip install llmcompressor`) — e.g. `QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])` then `oneshot(...)`. - **NVIDIA ModelOpt** (`pip install nvidia-modelopt`) — offline (pre-quantize once, fast startup, validated before deploy) vs. online (load BF16 + flag, convenient but high startup time/VRAM risk) modes. Pre-quantized: `--quantization modelopt_fp8` or `--quantization modelopt_fp4`. Create checkpoints via `hf_ptq.py` (`--qformat {fp8,nvfp4,nvfp4_mlp_only}`, `--kv_cache_qformat` default `fp8` — consider setting explicitly). SGLang ships `examples/usage/modelopt_quantize_and_export.py` for a full quantize+export workflow, plus a Python `ModelConfig(quantization=..., ...)` / `LoadConfig(modelopt_export_path=..., modelopt_checkpoint_save_path=...)` API for programmatic quantize/export/restore. Hopper+ hardware recommended; insufficient GPU memory triggers weight offloading and very long quantization time. - **ModelSlim** (Ascend-only, `msmodelslim`) — one-click `msmodelslim quant --model_path ... --quant_type w8a8 ...`; supported schemes include `W4A4_DYNAMIC`, `W8A8`, `W8A8_DYNAMIC` (linear); `W4A4_DYNAMIC`, `W4A4_MXFP4`, `W4A8_DYNAMIC`, `W4A8_MXFP`, `W8A8_DYNAMIC` (MoE); several others (`W4A8`, `W4A16` linear/MoE, KV Cache, Attention) are still TBD/in progress. ### Online quantization ```bash python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --quantization fp8 --port 30000 --host 0.0.0.0 ``` More online methods (`awq`, `gptq`, `marlin`, `gptq_marlin`, `awq_marlin`, `bitsandbytes`, `gguf`) are planned. - **`nvfp4_online`** — converts eligible BF16/FP16/FP8 MoE expert weights to NVFP4 at load time with **per-token** FP32 activation scales (use `modelopt_fp4` instead for serialized checkpoints / per-tensor scales). Weights: static E4M3 block scales + static per-tensor FP32 scale from weight amax (gated MoE w1/w3 pair shares one scale); FP8-checkpoint experts are dequantized then requantized to NVFP4 during load. Only `--moe-runner-backend flashinfer_trtllm` or `flashinfer_trtllm_routed` supported (defaults to `flashinfer_trtllm` if unset); disables shared-expert fusion (shared experts stay at checkpoint precision); TP-compatible (per-token activation scales computed locally per rank); `SGLANG_FP4_IGNORED_LAYERS` lets listed FP8 experts stay FP8 instead of converting. ```bash python3 -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \ --tp-size 2 --ep-size 2 --quantization nvfp4_online --port 30000 --host 0.0.0.0 ``` - **`quark_int4fp8_moe`** (AMD CDNA3/CDNA4) — MoE weights dynamically quantized to int4, upcast to FP8 at inference; other layers (e.g. attention projections) quantized online straight to FP8. - **`quark_mxfp4`** (AMD CDNA4, e.g. MI355x) — quantizes BF16 or NVFP4 weights to MXFP4 at load, dynamic MXFP4 activations, MXFP4 GEMMs. Also supports **NVFP4→MXFP4** requantization (reads quant metadata from `config.json`/`hf_quant_config.json`, respects producer-declared excluded modules, supports mixed-precision NVFP4 checkpoints) and **FP8→MXFP4** requantization (load FP8 → dequantize to BF16 → requantize to MXFP4, progressively during weight loading). - **`auto-round-int8`** (Intel Neural Compressor) — INT8 per-channel weight + INT8 per-token dynamic activation; validated on Intel Xeon Scalable and NVIDIA A100. - **Diffusion models on Ascend A5** (separate from LLM serving, via `sglang serve`/`sglang generate`): online `--quantization mxfp8` for FP16/BF16 transformer weights, or offline ModelSlim-quantized checkpoints auto-detected from `quant_model_description.json`. Requires Ascend A5, CANN ≥ 8.0.RC3. ### Reasoning-Aware Compression (RAC) Not quantization, but the doc set's other model-compression axis: **pruning** (weight sparsity via SparseGPT/Wanda) for reasoning models. Standard one-shot pruning minimizes `||WX - W'X||²` against calibration activations `X` built from *prompt* tokens — a reasonable proxy when prompts dominate token count. Reasoning models invert that ratio (thousands of self-generated chain-of-thought tokens per query), so prompt-only calibration optimizes pruned weights for a distribution the model barely runs in production, which doesn't degrade gracefully — the pruned model rambles (longer CoT, lower accuracy), so pruning can make the model **slower**, not faster. Example: 50%-sparsity C4-calibrated DeepSeek-R1-Distill-Qwen-7B took ~6x longer to evaluate MATH-500 than the dense model it was meant to accelerate. RAC's fix: calibrate on the model's own **on-policy rollout**, reconstructing prompt and decode activations jointly (`X_RAC = [X_prompt, X_decode]`) — a drop-in change to the SparseGPT/Wanda solver itself. Reference numbers (DeepSeek-R1-Distill-Qwen-7B, MATH-500, SparseGPT, 50% sparsity, 1M calibration tokens): | Calibration set | acc@1 | Eval wall clock | |---|---|---| | Dense (no pruning) | 0.936 | 23.3 min | | C4 | 0.744 | 135.0 min | | Task prompts only | 0.812 | 115.6 min | | **RAC (prompts + on-policy CoT)** | **0.900** | **35.3 min** | Across DeepSeek-R1-Distill-Qwen (1.5B–32B) and Qwen3 (1.7B–14B), RAC reportedly retains up to 95% of dense accuracy at 50% sparsity, beating prompt-only calibration by up to 17 accuracy points. Three-phase runnable recipe at `examples/usage/reasoning_aware_compression`: | Phase | Script | Role | |---|---|---| | I | `rac_collect_traces.py` | `sgl.Engine` samples on-policy CoT traces (SGLang's batched generation does the heavy lifting; paper's budget is 1M tokens) | | II | `rac_prune.py` | `llm-compressor` runs SparseGPT/Wanda on those activations (external dependency: `pip install "llmcompressor>=0.12.0"`) | | III | `rac_serve_and_eval.py` | Serves the sparse checkpoint with SGLang, scores MATH-500 | ```bash cd examples/usage/reasoning_aware_compression python rac_collect_traces.py --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --dataset open-r1/OpenR1-Math-220k --prompt-column problem --target-tokens 1000000 --output-dir ./rac_traces_math python rac_prune.py --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --calibration ./rac_traces_math/traces.jsonl --sparsity 0.5 --output-dir ./rac_pruned_50 python -m sglang.launch_server --model-path ./rac_pruned_50 ``` Always report **mean completion length** and **wall clock** alongside accuracy when comparing pruned reasoning checkpoints — accuracy alone hides the "rambling" failure mode. ## Key Parameters - `--quantization ` — online quantization method; omit entirely when loading an already-quantized checkpoint (unless forcing a specific SGLang kernel path like `w8a8_fp8`). - `--fp8-gemm-backend`, `--fp4-gemm-backend` — GEMM backend selection for quantized compute. - `--kv_cache_qformat` (ModelOpt `hf_ptq.py`) — separate from `--kv-cache-dtype` at serving time (see [[concepts/hierarchical-caching]]). - `SGLANG_USE_AITER`, `SGLANG_FP4_IGNORED_LAYERS`, `SGLANG_USE_CPU_ENGINE` — environment toggles for AMD acceleration, NVFP4 exclusions, and CPU serving. - RAC: `--target-tokens`, `--sparsity`, `--calibration` (RAC scripts, not SGLang server flags). ## When To Use - Prefer **offline** quantization for production — validated accuracy, fast startup, no runtime VRAM spike. - Use **online** quantization for quick experimentation or when no pre-quantized checkpoint exists, understanding the startup-time/VRAM tradeoff. - Use `nvfp4_online`/`quark_mxfp4` specifically to convert existing FP8/NVFP4/BF16 MoE checkpoints to a hardware-native format at load time without a separate offline pass. (`petit_nvfp4` is different: it's a serving/loading backend that enables NVFP4 on ROCm and is auto-selected when loading an already-NVFP4 checkpoint on AMD hardware — the docs don't describe it doing BF16/FP8→NVFP4 conversion the way `nvfp4_online`/`quark_mxfp4` do.) - Use RAC specifically when pruning a **reasoning** model (long CoT emitters) — standard prompt-only calibration is documented to actively backfire (slower AND less accurate) in this regime. ## Risks & Pitfalls - Passing `--quantization` alongside an already-quantized checkpoint enables online quantization *on top of* pre-quantized weights — the doc explicitly warns not to do this. - Marlin-based methods (`awq_marlin`, `gptq_marlin`) are CUDA-only; they silently aren't available on AMD/Ascend. - Mixed-bit quantization (different bit-widths within a fused layer, e.g. QKV) is not fully supported due to layer-fusion conflicts. - Quantized MoE models can hit kernel limitations (e.g. no `mlp.gate` layer quantization support) — the workaround is to skip quantizing those layers, not force it. - Some quantized-VLM format combinations silently produce near-zero accuracy (e.g. Qwen2.5-VL-7B with `auto_round:auto_gptq`) rather than erroring — always validate quantized models against benchmarks before deployment, as the docs explicitly recommend. - FP4 is not uniformly experimental — the compatibility matrix labels only AMD's `bitsandbytes` support "Experimental"; several FP4 methods (`mxfp4`, `modelopt_fp4`, `nvfp4_online`, `petit_nvfp4`) are documented as supported (not experimental) on their stated hardware. Don't assume a blanket experimental/accuracy-variance caveat applies across all FP4 methods. - RAC's lesson is specific to pruning **reasoning models**: calibrating a sparsity solver only on prompt-distribution data optimizes for a token distribution the model barely runs in production once it emits long chain-of-thought, which can quietly increase latency even at an acceptable-looking accuracy number. Always check mean completion length and wall clock, not accuracy alone, when pruning a reasoning model — the docs don't establish this as a general law for all pruning/compression calibration. ## Related Concepts - [[concepts/hierarchical-caching]] — KV-cache quantization (FP8/FP4) is a separate, complementary memory-reduction axis from the weight quantization covered here. - [[concepts/parallelism-and-disaggregation]] — MoE runner backends (`deep_gemm`, `flashinfer_cutlass`, `flashinfer_trtllm`, `flashinfer_mxfp4`) tie EP configuration to the quantization scheme in use. - [[concepts/speculative-decoding]] — `--speculative-draft-model-quantization` lets a draft model use independent quantization from the target model. - [[concepts/attention-backends-and-cuda-graph]] — some attention/CUDA-graph paths are quantization-format-specific (e.g. FP8/FP4 KV cache fusion). ## Sources - raw/github_doc-docs-docs-advanced-features-quantization-mdx.md - raw/github_doc-docs-docs-advanced-features-reasoning-aware-compression-mdx.md - raw/github_doc-docs-docs-advanced-features-quantized-kv-cache-mdx.md - raw/github_doc-docs-docs-advanced-features-expert-parallelism-mdx.md - raw/github_doc-docs-docs-advanced-features-speculative-decoding-mdx.md --- title: "References, FAQ, and Troubleshooting" type: concept tags: [api, operator, developer, advanced] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-references-overview-mdx.md", "raw/github_doc-docs-docs-references-faq-mdx.md", "raw/github_doc-docs-docs-references-environment-variables-mdx.md", "raw/github_doc-docs-docs-references-custom-chat-template-mdx.md", "raw/github_doc-docs-docs-references-production-metrics-mdx.md", "raw/github_doc-docs-docs-references-production-request-trace-mdx.md", "raw/github_doc-docs-docs-references-torch-compile-cache-mdx.md", "raw/github_doc-docs-docs-references-nightly-precision-regression-mdx.md", "raw/github_doc-docs-docs-references-post-training-integration-mdx.md", "raw/github_doc-docs-docs-references-frontend-choices-methods-mdx.md", "raw/github_doc-docs-docs-references-frontend-frontend-index-mdx.md", "raw/github_doc-docs-docs-references-frontend-frontend-tutorial-mdx.md", "raw/github_doc-docs-docs-references-multi-node-deployment-multi-node-index-.md", "raw/github_doc-docs-docs-references-multi-node-deployment-multi-node-mdx.md", "raw/github_release-v0-5-18.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition The references section is SGLang's catch-all for cross-cutting operational knowledge that doesn't belong to a single feature page: troubleshooting/FAQ, the environment-variable catalog, production observability (Prometheus metrics, OpenTelemetry tracing), custom chat templates, the frontend DSL's "choices" selection methods, post-training/RL integration, torch.compile caching, multi-node deployment patterns, and the internal nightly numerical-regression test framework (raw/github_doc-docs-docs-references-overview-mdx.md). ## How It Works ### Troubleshooting and FAQ (raw/github_doc-docs-docs-references-faq-mdx.md) - **CUDA OOM**: reduce `--chunked-prefill-size` (e.g. to 4096/2048) for prefill-time OOM; lower `--max-running-requests` for decode-time OOM; reduce `--mem-fraction-static` (e.g. to 0.8/0.7) to shrink the KV-cache pool generally (at a concurrency/throughput cost); or reduce `logprob_start_len` in sampling params if the OOM is from requesting input logprobs on a long prompt. - **"Illegal memory access" errors** are sometimes OOM misreported under a different name — apply the OOM guidance above before assuming it's a kernel bug. - **Server hangs** at init or during running usually trace to OOM (watch `avail mem` right after init), NCCL/network issues, or a genuine bug — the same three memory knobs above (`--mem-fraction-static`, `--cuda-graph-max-bs-decode`, `--chunked-prefill-size`) are the first things to try. - **Nondeterminism at temperature 0**: dynamic batching (~95% of the effect) and prefix caching (the rest) cause different batch sizes to dispatch different CUDA/cuBLAS kernels, producing small numerical differences that compound across layers. `--disable-radix-cache` plus sending one request at a time gets you close to deterministic. For a real fix, use `--enable-deterministic-inference` (see the [SGLang deterministic-inference blog post](https://lmsys.org/blog/2025-09-22-sglang-deterministic/)). ### Environment variables (raw/github_doc-docs-docs-references-environment-variables-mdx.md) The canonical prefix is `SGLANG_` — the legacy `SGL_` prefix is deprecated and auto-rewritten at import time with a warning (removal planned). A few vendor-prefixed variables keep their upstream names (`MOONCAKE_*`, `ASCEND_*`). Highlights by category: - **General**: `SGLANG_CACHE_DIR` (default `~/.cache/sglang`) is now the root for *all* compiled-kernel caches — Triton, Inductor, FlashInfer, the CUDA driver, and DeepGEMM nest under it unless their own env var (`TRITON_CACHE_DIR`, `TORCHINDUCTOR_CACHE_DIR`, `FLASHINFER_WORKSPACE_BASE`, `CUDA_CACHE_PATH`, `SGLANG_DG_CACHE_DIR`) is set explicitly — the v0.5.18 release notes confirm this consolidation shipped in that release and that the first launch after upgrading past it recompiles every cache once (raw/github_release-v0-5-18.md; see [[summaries/release-digest]]). `SGLANG_USE_MODELSCOPE`, `SGLANG_HEALTH_CHECK_TIMEOUT` (default 20s), `SGLANG_MAX_NEW_TOKENS_LIMIT` (server-side hard cap, unset by default). - **Performance tuning**: `SGLANG_SET_CPU_AFFINITY`, `SGLANG_ENABLE_TORCH_COMPILE`, `SGLANG_FLASHINFER_AUTOTUNE_CACHE` (persist FlashInfer autotune results across runs; `0` forces re-autotune each start), `SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD` (default 8192), `SGLANG_MAX_KV_CHUNK_CAPACITY` (default 131072, DeepSeek MHA chunked prefix cache), `SGLANG_USE_SGL_FA3_KERNEL` (default true). - **DeepGEMM**: `SGLANG_ENABLE_JIT_DEEPGEMM` (auto-on for SM90/SM100 with the package installed), `SGLANG_JIT_DEEPGEMM_PRECOMPILE`, `SGLANG_JIT_DEEPGEMM_COMPILE_WORKERS` (default 4), `SGLANG_DG_CACHE_DIR`, `SGLANG_JIT_DEEPGEMM_FAST_WARMUP` (the fetched docs report a reduction from a 30-minute warmup to under 3 minutes at some runtime cost — no model/hardware/kernel-set is specified, so treat this as a reported figure, not a portable benchmark). - **DeepEP / MoE dispatch**: `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` (default 128), `SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK` (default 1024), `SGLANG_ENABLE_MOE_DEFERRED_FINALIZE` (default true). - **MORI** (AMD EP communication): `SGLANG_MORI_DISPATCH_DTYPE` (`auto`/`bf16`/`fp8`/`fp4`), `SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK` (default 4096), `SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK` (default 128, must be ≥ worst-case per-rank forward tokens or the server refuses to start). - **DSA (DeepSeek V3.2-style sparse attention)**: `SGLANG_DSA_FUSE_TOPK` (default true; `SGLANG_NSA_FUSE_TOPK` is a deprecated alias), `SGLANG_DSA_TOPK_FLASHINFER_DETERMINISTIC`, `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD`. - **Ascend-specific env vars** are documented separately — see [[concepts/supported-hardware]]. ### Custom chat templates (raw/github_doc-docs-docs-references-custom-chat-template-mdx.md) This is the **server-side** chat template (`conversation.py`) used by the OpenAI-compatible API — distinct from the SGLang frontend-language chat template (`lang/chat_template.py`). By default the server uses the template baked into the HF tokenizer, which "just works" for most official models. Override with `--chat-template `, either a built-in name (e.g. `llama-2`), a JSON file matching the `conversation.py` schema (`name`/`system`/`user`/`assistant`/`sep_style`/`sep`/`stop_str`), or a raw Jinja2 file in the HF chat-templating format. ### Production observability **Metrics** (raw/github_doc-docs-docs-references-production-metrics-mdx.md): `--enable-metrics` exposes a Prometheus endpoint (`/metrics`) with per-model-name-labeled counters/gauges/histograms — `sglang:prompt_tokens_total`, `sglang:generation_tokens_total`, `sglang:cache_hit_rate`, `sglang:time_to_first_token_seconds`, `sglang:e2e_request_latency_seconds`, `sglang:time_per_output_token_seconds`, `sglang:num_running_reqs`, `sglang:gen_throughput`, `sglang:spec_num_steps`/`spec_num_draft_tokens`. `--enable-mfu-metrics` additionally exports `sglang:estimated_flops_per_gpu_total` and `sglang:estimated_{read,write}_bytes_per_gpu_total` (cumulative counters; use `rate(...)` in PromQL) for approximate Model-FLOPs-Utilization and memory-bandwidth signals. A prebuilt Grafana dashboard + Docker Compose stack lives under `examples/monitoring/`. **Distributed tracing** (raw/github_doc-docs-docs-references-production-request-trace-mdx.md): `--enable-trace --otlp-traces-endpoint ` exports OpenTelemetry spans to a collector (Jaeger for visualization). `SGLANG_TRACE_LEVEL` (0=off, 1=important slices, 2=all non-nested slices, 3=everything, default) is settable at startup or dynamically via `curl .../set_trace_level?level=N` — but only if `--enable-trace` was passed at launch. **Async tracing** (`SGLANG_TRACE_ASYNC=1`) offloads span creation to a dedicated per-worker exporter process over ZMQ to avoid OTel's synchronous locking overhead at high batch sizes, while preserving the same exported span tree via pre-generated span IDs. The three-level context model is `TraceReqContext` → `TraceThreadContext` → `TraceSliceContext`. ### Post-training / RL integration (raw/github_doc-docs-docs-references-post-training-integration-mdx.md) SGLang is positioned as the de facto rollout/inference backend for RLHF and post-training frameworks, citing adoption by **Miles**, **slime** (used to train GLM-4.6), **AReaL**, **ROLL**, **verl**, **Unsloth**, **LLaMA Factory**, **Tunix**, and **RL2**. The capabilities cited as load-bearing for this role: refit for colocated or disaggregated training/rollout, deferrable generation for partial rollouts, fine-grained engine sleep/wake for maximum-power rollout vs. training phases, training/serving numerical alignment, cache-aware load-balancing routing, and deterministic inference (zero KL divergence between rollout and training policies). ### torch.compile cache portability (raw/github_doc-docs-docs-references-torch-compile-cache-mdx.md) SGLang runs `torch.compile` in `max-autotune-no-cudagraphs` mode, whose autotuning is slow. To skip recompilation across a fleet of identical machines: generate the cache once with `TORCHINDUCTOR_CACHE_DIR=/root/inductor_root_cache python3 -m sglang.launch_server --model --enable-torch-compile`, then copy that directory to other machines and launch with the same env var set. ### Frontend DSL choices methods (raw/github_doc-docs-docs-references-frontend-choices-methods-mdx.md, .../frontend-frontend-index-mdx.md, .../frontend-frontend-tutorial-mdx.md) The frontend language's `sgl.gen(..., choices=[...])` construct (only on the `RuntimeEndpoint` backend — `OpenAI` and other backends have their own bespoke selection) supports three `choices_method` values: **`token_length_normalized`** (default — highest average logprob across all of an option's tokens; can fail when one option has many tokens whose later tokens are predicted confidently given the earlier ones, e.g. `["Paris", "Antidisestablishmentarianism"]`), **`greedy_token_selection`** (compares only the first token's logprob, extending shorter overlapping options by their average logprob for a fair comparison against longer ones; can be misled by an attractive-but-wrong initial token, e.g. picking "Donald Duck" over "Millard Fillmore" for "name a US president"), and **`unconditional_likelihood_normalized`** (normalizes by each option's *unconditional* token logprobs, per an EleutherAI method — costs one extra LLM call). The frontend tutorial itself demonstrates the `@function`/`gen`/`fork`/`regex`/`run_batch`/`stream` primitives — see [[concepts/sglang-overview]] for where the frontend DSL sits relative to the runtime and server API surfaces. ### Multi-node deployment (raw/github_doc-docs-docs-references-multi-node-deployment-multi-node-index-.md, .../multi-node-deployment-multi-node-mdx.md) Beyond a single node, `--nnodes`/`--node-rank`/`--dist-init-addr` split TP across machines (e.g. Llama 3.1 405B fp16 across two 8-GPU nodes with `--tp 16`; the fp8 build fits on one 8-GPU node with `--tp 8`). SLURM job scripts follow the same pattern, one task per node. Detailed multi-node PD-disaggregation deployment (Kubernetes, LeaderWorkerSet, RBG) and the large-scale DeepSeek/Kimi K2 EP write-ups live under a dedicated `multi_node_deployment/` doc subtree — that material overlaps with [[concepts/parallelism-and-disaggregation]] and is not duplicated here. ### Nightly precision regression testing (raw/github_doc-docs-docs-references-nightly-precision-regression-mdx.md) An internal CI framework (not a user-facing feature) that detects silent numerical regressions by comparing per-layer hidden states between consecutive nightly runs on 8×H200. It uses a **rolling baseline**: dump per-layer hidden states (strided capture — layer 0, last layer, every 8th layer between), compare against the last baseline with a tensor comparator (`rel_diff` threshold, default `1e-3`), and if it passes, the new run becomes the baseline; baselines persist across ephemeral CI runners via a required HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`/`SGLANG_PRECISION_HF_TOKEN` — there is no local-only fallback). A `capture_signature` hash (schema version + TP size + dumper filter + max_tokens/ignore_eos) ensures baselines are only compared against a matching capture shape. Known limitation: because the baseline rolls forward on every pass, tiny per-run differences can accumulate ("baseline drift") and eventually mask a real regression or trigger a false failure. ## Key Parameters - **`--enable-deterministic-inference`** — the real fix for temperature-0 nondeterminism (vs. the `--disable-radix-cache` + single-request workaround). - **`--chat-template`** — JSON or Jinja override for the server-side (OpenAI-API) chat template; distinct from the frontend-language template. - **`--enable-metrics` / `--enable-mfu-metrics` / `--enable-trace` / `--otlp-traces-endpoint`** — the three production-observability switches. - **`SGLANG_TRACE_LEVEL`, `SGLANG_TRACE_ASYNC`** — tracing verbosity and the async-exporter opt-in for high-batch-size deployments. - **`SGLANG_CACHE_DIR`** — single root for all compiled-kernel caches as of v0.5.18. - **`choices_method`** — `token_length_normalized` (default) / `greedy_token_selection` / `unconditional_likelihood_normalized` for frontend-DSL `choices=[...]` selection. ## When To Use Reach for this page's FAQ section first for OOM/hang/nondeterminism symptoms — the docs walk through a handful of memory and caching flags for these. But hangs specifically can also stem from NCCL/network errors or other SGLang bugs, not just memory pressure — the docs direct you to file a GitHub issue in those cases, so don't treat the flag checklist as exhaustive before escalating. Use the environment-variable catalog when a server-arguments flag doesn't cover the tuning knob you need (many performance and MoE-dispatch controls are env-var-only, not CLI flags). Enable `--enable-metrics`/`--enable-trace` for any production deployment that needs SLA visibility; reach for the async tracing mode specifically once synchronous OTel spans start measurably hurting throughput at high concurrency. Use the torch.compile cache-copying trick whenever deploying `--enable-torch-compile` across more than one identical machine, to avoid paying the autotune cost per box. ## Risks & Pitfalls - `--otlp-traces-endpoint` alone does nothing without `--enable-trace`; the dynamic `/set_trace_level` endpoint also has no effect unless `--enable-trace` was passed at startup. - The v0.5.18 cache-directory consolidation (`SGLANG_CACHE_DIR`) means the *first* launch after upgrading past that version recompiles every cache once — the release notes recommend copying or symlinking pre-warmed/volume-mounted cache directories to the new locations (`deep_gemm` and `flashinfer` are called out as the expensive ones to rebuild) (raw/github_release-v0-5-18.md; see [[summaries/release-digest]]). - MFU-related metrics (`sglang:estimated_*`) are modeled estimates, not direct hardware counters — treat them as trend signals, not ground truth. - The nightly-precision rolling baseline can silently drift over weeks/months; a "PASSED" status only means "no sudden change since yesterday," not "matches the original golden reference." - `SGL_*` environment variables worked (auto-rewritten to `SGLANG_*` with a deprecation warning) in the docs snapshot fetched 2026-08-24, but removal is planned — new configuration should use `SGLANG_*` rather than relying on the alias's permanence. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/developer-and-benchmarking]] - [[concepts/supported-hardware]] - [[concepts/parallelism-and-disaggregation]] - [[summaries/release-digest]] ## Sources - raw/github_doc-docs-docs-references-overview-mdx.md - raw/github_doc-docs-docs-references-faq-mdx.md - raw/github_doc-docs-docs-references-environment-variables-mdx.md - raw/github_doc-docs-docs-references-custom-chat-template-mdx.md - raw/github_doc-docs-docs-references-production-metrics-mdx.md - raw/github_doc-docs-docs-references-production-request-trace-mdx.md - raw/github_doc-docs-docs-references-torch-compile-cache-mdx.md - raw/github_doc-docs-docs-references-nightly-precision-regression-mdx.md - raw/github_doc-docs-docs-references-post-training-integration-mdx.md - raw/github_doc-docs-docs-references-frontend-choices-methods-mdx.md - raw/github_doc-docs-docs-references-frontend-frontend-index-mdx.md - raw/github_doc-docs-docs-references-frontend-frontend-tutorial-mdx.md - raw/github_doc-docs-docs-references-multi-node-deployment-multi-node-index-.md - raw/github_doc-docs-docs-references-multi-node-deployment-multi-node-mdx.md - raw/github_release-v0-5-18.md --- title: "Router and Model Gateway" type: concept tags: [architecture, api, parallelism, operator, well-established] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md", "raw/github_doc-docs-docs-advanced-features-llm-d-mdx.md", "raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md", "raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition SGLang has two distinct answers to "how do I route across multiple SGLang replicas or models," operating at different layers: 1. **SGLang Model Gateway** (formerly "Router") — an in-repo, Rust-based routing/load-balancing process (`sgl-model-gateway` / `sglang_router`) that sits directly in front of a fleet of SGLang worker processes, deeply integrated with the SGLang runtime (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md). 2. **llm-d** — a Kubernetes-native, engine-agnostic fleet-scale inference framework (a CNCF Sandbox project) that coordinates many SGLang (or mixed SGLang/vLLM) instances across a cluster, adding prefix-aware routing, distributed KV-cache management, and autoscaling on top of whatever routing SGLang itself does (raw/github_doc-docs-docs-advanced-features-llm-d-mdx.md). ## How It Works ### SGLang Model Gateway architecture **Control plane:** - **Worker Manager** — discovers capabilities (`/get_server_info`, `/get_model_info`), tracks load, registers/removes workers. - **Job Queue** — serializes add/remove requests; status via `GET /workers/{worker_id}`. - **Load Monitor** — feeds cache-aware and power-of-two policies live load stats. - **Health Checker** — probes workers, updates readiness and circuit-breaker state. - **Tokenizer Registry** — dynamically registered tokenizers, async-loaded from HuggingFace or local paths. **Data plane:** - **HTTP routers** (regular & PD) implement `/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify`, `/v1/tokenize`, `/v1/detokenize`. - **gRPC router** — a fully in-Rust pipeline (tokenizer, reasoning parser, tool parser all in-process) for the highest throughput; supports single-stage and PD topologies, including embeddings/classification. - **OpenAI router** — proxies OpenAI-compatible endpoints to external vendors while keeping chat history and multi-turn orchestration local. **Storage/privacy:** conversation and `/v1/responses` history is stored at the router tier (`memory`, `none`, Oracle ATP, PostgreSQL, or Redis), so the same context can be reused across models/MCP loops without leaking data upstream (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md). ### Deployment modes ```bash Command # Co-launch router + worker fleet in one process python -m sglang_router.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --dp-size 4 --host 0.0.0.0 --port 30000 # Separate launch: workers independent, router points at their HTTP endpoints python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8000 python -m sglang_router.launch_router --worker-urls http://worker1:8000 http://worker2:8001 --policy cache_aware # gRPC (highest throughput, native reasoning/tool pipelines) python -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --grpc-mode --port 20000 python -m sglang_router.launch_router --worker-urls grpc://127.0.0.1:20000 --model-path meta-llama/Llama-3.1-8B-Instruct \ --reasoning-parser deepseek-r1 --tool-call-parser json --host 0.0.0.0 --port 8080 # PD disaggregation routing python -m sglang_router.launch_router --pd-disaggregation \ --prefill http://prefill1:30001 9001 --decode http://decode1:30011 \ --prefill-policy cache_aware --decode-policy power_of_two # OpenAI backend proxy (keeps history/MCP local) python -m sglang_router.launch_router --backend openai --worker-urls https://api.openai.com --history-backend memory # Multi-Model Inference Gateway (IGW) mode ./target/release/sgl-model-gateway --enable-igw --policy cache_aware --max-concurrent-requests 512 ``` (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md) Note the general recommendation elsewhere in the docs: for DP throughput, "always favor data parallelism... Refer to SGLang Model Gateway (former Router) for a better data parallelism rather than using `dp_size` parameter" (raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md) — the router is the recommended way to scale DP, not `launch_server --dp-size` alone. ### Load balancing policies | Policy | Description | |---|---| | `random` | Uniform random | | `round_robin` | Cycles through workers | | `power_of_two` | Samples two workers, picks the lighter one | | `cache_aware` (**default**) | Combines RadixAttention prefix-cache locality with load balancing | | `bucket` | Divides workers into dynamic load buckets | `cache_aware` tuning: `--cache-threshold` (default `0.3`, minimum prefix-match ratio for a cache hit), `--balance-abs-threshold` (`64`), `--balance-rel-threshold` (`1.5`), `--eviction-interval-secs` (`120`), `--max-tree-size` (`67108864` nodes) (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md). ### Reliability and flow control - **Retries** — exponential backoff on retryable status codes `408, 429, 500, 502, 503, 504`. - **Circuit breaker** (per worker) — states `Closed` → `Open` → `Half-Open`; tuned via `--cb-failure-threshold` (`5`), `--cb-success-threshold` (`2`), `--cb-timeout-duration-secs` (`30`), `--cb-window-duration-secs` (`60`); disable with `--disable-circuit-breaker`. - **Rate limiting & queuing** — `--max-concurrent-requests`, `--rate-limit-tokens-per-second`, `--queue-size`, `--queue-timeout-secs`; over-limit requests FIFO-queue, then `429` (queue full) or `408` (queue timeout). - **Health checks** — `--health-check-interval-secs` (`30`), `--health-check-timeout-secs` (`10`), `--health-success-threshold`/`--health-failure-threshold`, `--health-check-endpoint`. (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md) ### Kubernetes service discovery ```bash Command python -m sglang_router.launch_router --service-discovery --selector app=sglang-worker role=inference \ --service-discovery-namespace production --service-discovery-port 8000 # PD mode discovery --pd-disaggregation --prefill-selector app=sglang component=prefill --decode-selector app=sglang component=decode --service-discovery ``` Prefill pods can expose bootstrap ports via the `sglang.ai/bootstrap-port` annotation; RBAC needs `get`/`list`/`watch` on pods (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md). ### Reasoning/tool parsing and tokenizer management The gRPC router natively detects and separates reasoning blocks (`--reasoning-parser`: `deepseek-r1`, `qwen3`, `qwen3-thinking`, `kimi`, `glm45`, `step3`, `minimax`) and parses tool calls (`--tool-call-parser`: `json`, `python`, `xml`) in-process, with incremental streaming-safe parsing. Tokenizers can be HuggingFace, local, or Tiktoken-auto-detected, with a two-level cache: L0 (exact-match, whole-string) and L1 (prefix-match) (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md). ### Production / high-availability guidance The gateway supports multiple replicas behind a load balancer, but **state is not shared across gateway replicas**: | Component | Shared across replicas? | Impact | |---|---|---| | Worker Registry | No | Each replica discovers workers independently | | Radix Cache Tree | No | Cache hit rate drops an estimated **10–20%** | | Circuit Breaker state | No | Each replica tracks failures independently | | Rate limiting | No | Limits apply per-replica, not globally | Recommendations: prefer many small gateway replicas over one large instance; use Kubernetes service discovery; accept the cache-efficiency tradeoff, or configure load-balancer session affinity (consistent hash on user ID/API key) if cache hit rate is critical. Security checklist for production: enable TLS (`--tls-cert-path`, `--tls-key-path`, etc.) and mTLS for worker communication on untrusted networks, set `--api-key`, use a secrets manager, rotate certs/keys, restrict network access (raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md). ### llm-d: fleet-scale routing on Kubernetes A single SGLang server already maximizes cache reuse within its own replica via RadixAttention, but **across replicas**, cache locality breaks under naive round-robin: related requests scatter, radix-cache hit rates collapse, TTFT inflates on long prompts, and accelerators sit underused. llm-d adds a cluster-level layer: - **Prefix-aware routing** — the llm-d Router scores each replica on prefix-cache locality *and* current load (SGLang publishes KV-cache events the router subscribes to), instead of round-robin — raising RadixAttention hit rates on multi-turn/shared-prefix workloads. - **Distributed KV-cache management** — a global index tracks which token blocks live on which replica; tiered offloading spills cache to CPU memory or local SSD. - **Prefill/decode disaggregation** — prompt processing and token generation on separate workers, KV-cache moved over high-speed interconnects. - **SLO-aware autoscaling and flow control** — scales SGLang pools on queue depth/demand signals rather than raw GPU utilization, with multi-tenant fairness. - **One control plane for mixed fleets** — SGLang and vLLM pools behind the same gateway, policies, and observability. llm-d builds on the Gateway API Inference Extension (standard Kubernetes `Gateway`/`HTTPRoute`/`InferencePool` resources), working with Istio, GKE Inference Gateway, and agentgateway. It's founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, and is a CNCF Sandbox project. Reported benchmark: prefix-aware routing delivered **3x higher output throughput and 2x faster TTFT** than round-robin on Llama 3.1 70B. SGLang is supported across llm-d's well-lit paths (intelligent scheduling, precise prefix-cache routing, tiered KV-cache management, PD disaggregation, flow control, autoscaling) — with one gap noted in the doc snapshot fetched 2026-08-24: **Multi-Node Wide Expert Parallelism is vLLM-specific "today"**, not yet available for SGLang under llm-d — the source itself calls this out as a moving target and points readers to the llm-d documentation for the latest per-engine support status (raw/github_doc-docs-docs-advanced-features-llm-d-mdx.md). ### Choosing between the two They're complementary, not competing: the SGLang Model Gateway is the close-coupled, single-cluster/single-deployment router with deep SGLang-specific integration (gRPC native pipeline, reasoning/tool parsing, conversation history); llm-d is the Kubernetes-native fleet orchestration layer for scaling *many* SGLang (and/or vLLM) deployments with cluster-wide prefix-aware routing and autoscaling. A common pattern per the llm-d doc: start with prefix-aware routing over an existing SGLang pool, then layer in the rest as bottlenecks appear. ## Key Parameters - `--policy` — load balancing policy (`cache_aware` default) - `--worker-urls` — HTTP/gRPC worker endpoints - `--pd-disaggregation`, `--prefill`, `--decode`, `--prefill-policy`, `--decode-policy` — PD-aware routing - `--enable-igw` — multi-model Inference Gateway mode - `--cache-threshold`, `--balance-abs-threshold`, `--balance-rel-threshold` — cache-aware policy tuning - `--cb-failure-threshold`, `--cb-timeout-duration-secs` — circuit breaker tuning - `--max-concurrent-requests`, `--rate-limit-tokens-per-second`, `--queue-size` — flow control - `--service-discovery`, `--selector`, `--prefill-selector`, `--decode-selector` — Kubernetes discovery - `--history-backend` — `memory`, `none`, `oracle`, `postgres`, `redis` ## When To Use The docs explicitly recommend the gateway over raw `--dp-size` specifically when favoring data parallelism for throughput ("when there is enough GPU memory, always favor data parallelism for throughput... for a better data parallelism rather than using `dp_size` parameter"). Beyond that sourced case, reaching for the gateway for any multi-worker deployment (PD-split, multi-model fleet) is reasonable operator guidance extending the same logic, not a claim the docs state directly. Reach for llm-d additionally when operating at Kubernetes fleet scale, needing cluster-wide prefix-aware routing across many pods, mixed SGLang/vLLM fleets, or SLO-driven autoscaling. ## Risks & Pitfalls - Gateway state (radix cache tree, circuit breaker, rate limits) is **per-replica**, not shared — horizontally scaling the gateway itself for HA is documented as an expected tradeoff, with cache hit rate "may decrease" / expected to fall by an estimated 10–20%. Session affinity is offered as an optional mitigation when cache efficiency is critical, not a guarantee that eliminates the loss. - Production security is opt-in: TLS/mTLS/API key must be explicitly configured; there's no secure-by-default posture called out in the source. - llm-d does not support Multi-Node Wide Expert Parallelism for SGLang in the doc snapshot fetched 2026-08-24 (vLLM-specific "today," per the source's own wording) — the source itself flags this as a moving target and points to the llm-d docs for the latest per-engine status, so don't treat it as a durable v0.5.18 limitation. - The `total_tokens` DP load-balance method (`--load-balance-method`, set on `launch_server` itself) only works when DP attention is enabled — it's a different, more limited mechanism than the gateway's `cache_aware` policy (raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md). ## Related Concepts - `[[concepts/architecture-and-radixattention]]` — the per-replica RadixAttention cache that `cache_aware` routing is built on top of, and reasons for its cross-replica degradation - `[[concepts/server-arguments]]` — `launch_server`-side flags (distinct from router-side flags documented here) - `[[concepts/parallelism-and-disaggregation]]` (planned) — deeper TP/DP/EP/PD treatment, including PD bootstrap mechanics ## Sources - raw/github_doc-docs-docs-advanced-features-sgl-model-gateway-mdx.md - raw/github_doc-docs-docs-advanced-features-llm-d-mdx.md - raw/github_doc-docs-docs-advanced-features-hyperparameter-tuning-mdx.md - raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md --- title: "Sampling Parameters" type: concept tags: [basic-usage, api, foundational, user] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition Sampling parameters are the set of controls that shape how SGLang's `/generate` endpoint turns model logits into output tokens — request framing (prompt/input format), core sampling knobs (temperature, top-p, top-k, min-p), repetition penalizers, constrained-decoding grammars, and miscellaneous generation options. The `/generate` endpoint is described as "the low-level endpoint of the runtime"; for a higher-level endpoint that automatically handles chat templates, the docs point to the [[concepts/server-apis|OpenAI-compatible API]] instead (raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md). ## How It Works The `/generate` endpoint's request body is defined by `GenerateReqInput` (in `io_struct.py`); `sampling_params` within that body is itself defined by `SamplingParams` (in `sampling_params.py`). Both are defined in source code — the docs note "you can also read the source code to find more arguments," which is a pointer to incomplete documentation rather than a stated extensibility guarantee. ### Request-level fields (outside `sampling_params`) | Argument | Type/Default | Description | |---|---|---| | `text` | `Optional[Union[List[str], str]] = None` | Input prompt(s). | | `input_ids` | `Optional[Union[List[List[int]], List[int]]] = None` | Token IDs instead of text. | | `input_embeds` | `Optional[Union[List[List[List[float]]], List[List[float]]]] = None` | Precomputed embeddings for input. | | `image_data` | `Optional[...] = None` | Raw image (PIL/path/URL/base64), processor output, or precomputed embedding — single image, list, or list of lists. | | `audio_data` | `Optional[Union[List[AudioDataItem], AudioDataItem]] = None` | Audio file name, URL, or base64 string. | | `sampling_params` | `Optional[Union[List[Dict], Dict]] = None` | The sampling controls detailed below. | | `rid` | `Optional[Union[List[str], str]] = None` | Request ID. | | `return_logprob` | `Optional[Union[List[bool], bool]] = None` | Return token log-probabilities. | | `logprob_start_len` | `Optional[Union[List[int], int]] = None` | Prompt offset from which to return logprobs; default `-1` = output tokens only. | | `top_logprobs_num` | `Optional[Union[List[int], int]] = None` | Number of top logprobs per position. | | `token_ids_logprob` | `Optional[Union[List[List[int]], List[int]]] = None` | Specific token IDs to return logprobs for. | | `return_text_in_logprobs` | `bool = False` | Detokenize tokens in returned logprobs. | | `stream` | `bool = False` | Stream output. | | `lora_path` | `Optional[Union[List[Optional[str]], Optional[str]]] = None` | Path to a LoRA adapter. | | `custom_logit_processor` | `Optional[Union[List[Optional[str]], str]] = None` | Serialized `CustomLogitProcessor` (via `.to_str()`). | | `return_hidden_states` | `Union[List[bool], bool] = False` | Return hidden states. | | `return_routed_experts` | `bool = False` | Return MoE routing data; requires `--enable-return-routed-experts`. Base64-encoded int32, shape `[num_tokens, num_layers, top_k]`; default returns the full sequence `[0, seqlen-1)` (RL workflows need the full sequence). | | `routed_experts_start_len` | `int = 0` | Absolute start offset for `return_routed_experts`; must be in `[0, prompt_tokens]`. Useful in multi-turn RL rollouts to skip re-transferring already-collected routing for earlier turns. | ### Sampling defaults source By default (`--sampling-defaults model`, the default), SGLang initializes unset sampling parameters from the model's own `generation_config.json`. Pass `--sampling-defaults openai` at launch to use SGLang/OpenAI constant defaults instead. Any parameter can always be overridden per-request via `sampling_params`. ```bash python -m sglang.launch_server --model-path --sampling-defaults model python -m sglang.launch_server --model-path --sampling-defaults openai ``` ### Core parameters | Argument | Type/Default | Description | |---|---|---| | `max_new_tokens` | `int = 128` | Max output length in tokens. | | `stop` | `Optional[Union[str, List[str]]] = None` | Stop word(s); generation halts if sampled. | | `stop_token_ids` | `Optional[List[int]] = None` | Stop word(s) as token IDs. | | `stop_regex` | `Optional[Union[str, List[str]]] = None` | Stop on regex match. | | `temperature` | `float` (model default; fallback `1.0`) | `0` = greedy sampling (does not by itself guarantee deterministic/reproducible output — see Risks & Pitfalls); higher = more diversity. | | `top_p` | `float` (model default; fallback `1.0`) | Samples from the smallest token set whose cumulative probability exceeds `top_p`; `1` = unrestricted. | | `top_k` | `int` (model default; fallback `-1`) | Randomly selects from the `k` highest-probability tokens. | | `min_p` | `float` (model default; fallback `0.0`) | Samples from tokens with probability `> min_p * highest_token_probability`. | ### Penalizers | Argument | Type/Default | Description | |---|---|---| | `frequency_penalty` | `float = 0.0` | Range `[-2, 2]`; penalizes tokens by frequency-so-far. Scaling grows linearly per occurrence. Negative encourages repetition, positive discourages it. | | `presence_penalty` | `float = 0.0` | Range `[-2, 2]`; penalizes tokens that have appeared at all so far. Scaling is constant once a token has occurred (unlike frequency_penalty). | | `repetition_penalty` | `float = 1.0` | Range `(0, 2]`; scales logits of previously generated tokens — `>1` discourages, `<1` encourages repetition; `1.0` is a no-op. | | `min_new_tokens` | `int = 0` | Forces at least this many tokens before a stop word/EOS can end generation; can misbehave if the token distribution is heavily skewed toward stop tokens. | ### Constrained decoding | Argument | Type/Default | Description | |---|---|---| | `json_schema` | `Optional[str] = None` | JSON schema constraint. | | `regex` | `Optional[str] = None` | Regex constraint. | | `ebnf` | `Optional[str] = None` | EBNF grammar constraint. | | `structural_tag` | `Optional[str] = None` | Structural tag constraint. | Only one of `json_schema` / `regex` / `ebnf` may be set per request. SGLang supports two grammar backends: **XGrammar** (default; JSON schema, regex, and EBNF, using the GGML BNF format) and **Outlines** (JSON schema and regex only), selected at launch with `--grammar-backend [xgrammar|outlines]`. Note: the cited source is internally inconsistent about regex support — its backend-inventory section lists regex under both XGrammar and Outlines, but its worked example labels the regex case "(Outlines backend only)" and the EBNF case "(XGrammar backend only)." Verify current regex-backend support against the runtime/backend documentation rather than relying on either statement alone. Full detail is in [[concepts/structured-outputs-and-tool-calling]]. ### Other options | Argument | Type/Default | Description | |---|---|---| | `n` | `int = 1` | Number of output sequences per request. Generating multiple outputs via `n > 1` is discouraged; repeating the prompt in separate requests offers better control and efficiency. | | `ignore_eos` | `bool = False` | Don't stop generation on EOS. | | `skip_special_tokens` | `bool = True` | Remove special tokens during decoding. | | `spaces_between_special_tokens` | `bool = True` | Add spaces between special tokens during detokenization. | | `no_stop_trim` | `bool = False` | Don't trim stop words/EOS from generated text. | | `custom_params` | `Optional[List[Optional[Dict[str, Any]]]] = None` | Parameters passed through to a `CustomLogitProcessor`. Note: the cited source's own type table (list-of-dicts) and its worked example (a single dict, not a list) disagree on the exact shape — treat this as unresolved rather than certain. | ### Worked examples **Basic request:** ```python import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "sampling_params": {"temperature": 0, "max_new_tokens": 32}, }, ) ``` **Multimodal request** (image data alongside chat-formatted text): ```python response = requests.post( "http://localhost:30000/generate", json={ "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n" "<|im_start|>user\n\nDescribe this image in a very short sentence.<|im_end|>\n" "<|im_start|>assistant\n", "image_data": "example_image.png", "sampling_params": {"temperature": 0, "max_new_tokens": 32}, }, ) ``` `image_data` accepts a file name, URL, or base64-encoded string. **Structured JSON output:** ```python import json, requests json_schema = json.dumps({ "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], }) response = requests.post( "http://localhost:30000/generate", json={ "text": "Here is the information of the capital of France in the JSON format.\n", "sampling_params": {"temperature": 0, "max_new_tokens": 64, "json_schema": json_schema}, }, ) ``` **Custom logit processor** (requires launching with `--enable-custom-logit-processor`): ```python from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor class DeterministicLogitProcessor(CustomLogitProcessor): def __call__(self, logits, custom_param_list): assert logits.shape[0] == len(custom_param_list) for i, param_dict in enumerate(custom_param_list): logits[i, :] = -float("inf") logits[i, param_dict["token_id"]] = 0.0 return logits ``` ```python response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "custom_logit_processor": DeterministicLogitProcessor().to_str(), "sampling_params": { "temperature": 0.0, "max_new_tokens": 32, "custom_params": {"token_id": 5}, }, }, ) ``` The same custom logit processor can be invoked via the OpenAI-compatible chat completions API using `extra_body={"custom_logit_processor": ..., "custom_params": ...}` — see [[concepts/server-apis]]. ## Key Parameters See the full tables above. The parameters most commonly tuned per-request are `temperature`, `top_p`, `top_k`, `max_new_tokens`, `stop`/`stop_token_ids`, `frequency_penalty`/`presence_penalty`/`repetition_penalty`, and the constrained-decoding trio `json_schema`/`regex`/`ebnf`. ## When To Use - Use the request-level fields (`text`/`input_ids`/`input_embeds`, `image_data`, `audio_data`) to shape what goes into the model — text-only, multimodal, or precomputed-embedding input. - Use core parameters (`temperature`, `top_p`, `top_k`, `min_p`) to trade off greedy vs. diverse output; `temperature=0` for greedy sampling. Greedy sampling alone does not guarantee bit-identical/deterministic output across runs — dynamic batching and cache paths can still select different kernels; true determinism additionally requires `--enable-deterministic-inference` (see [[concepts/observability-and-determinism]] and [[concepts/references-and-faq]]). - Use penalizers when a model repeats itself or over-uses certain tokens. - Use constrained decoding (`json_schema`/`regex`/`ebnf`) when output must conform to a fixed structure — see [[concepts/structured-outputs-and-tool-calling]]. - Use `custom_logit_processor` + `custom_params` for sampling logic not covered by the built-in parameters. - Prefer `--sampling-defaults openai` when you want stable, well-known defaults across different models rather than each model's own `generation_config.json` values. ## Risks & Pitfalls - `n > 1` (multiple outputs per request) is explicitly discouraged by the docs in favor of repeating the prompt across separate requests, for better control and efficiency. - `min_new_tokens` forcing generation past a natural stop point "might lead to unintended behavior" when the output distribution is heavily skewed toward stop/EOS tokens. - Only one of `json_schema`, `regex`, or `ebnf` can be set per request — combining them is not supported. - The Outlines grammar backend does not support EBNF; the XGrammar backend does not use standard BNF (it uses "GGML BNF format") — grammar syntax is backend-specific. - `logprob_start_len` defaults to `-1` (output tokens only) — requesting logprobs over the prompt itself requires explicitly setting a different start offset. - Sampling defaults are pulled from `generation_config.json` unless `--sampling-defaults openai` is set at launch — a value considered "unset" by the caller may still take on a per-model default silently. - `return_routed_experts` requires the server to have been launched with `--enable-return-routed-experts`; requesting it without that flag will not work. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/sending-requests]] - [[concepts/server-apis]] - [[concepts/offline-engine]] - [[concepts/structured-outputs-and-tool-calling]] - [[concepts/lora-and-model-loading]] ## Sources - raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md --- title: "Sending Requests" type: concept tags: [basic-usage, api, foundational, user] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-basic-usage-send-request-mdx.md", "raw/github_doc-docs-docs-basic-usage-offline-engine-api-mdx.md", "raw/github_doc-docs-docs-get-started-quickstart-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition Sending a request to SGLang is the act of getting a running server (launched per [[concepts/installation]]) to generate text, given a prompt, over HTTP — via cURL, plain Python `requests`, the OpenAI Python client, or SGLang's own native `/generate` endpoint. It is the fundamental request/response lifecycle for generation requests against a running server (raw/github_doc-docs-docs-basic-usage-send-request-mdx.md); note that SGLang also offers an in-process `Engine` with no HTTP server at all (see [[concepts/offline-engine]]), so this lifecycle does not underlie every SGLang usage pattern. ## How It Works ### 1. Launch a server ```python from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, terminate_process server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \ --host 0.0.0.0 --log-level warning """ ) wait_for_server(f"http://localhost:{port}") ``` Once running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `/redoc` (ReDoc), and `/openapi.json` (OpenAPI spec — useful for AI agents); substitute the actual port if different (raw/github_doc-docs-docs-basic-usage-send-request-mdx.md). ### 2. Send a request — four request/client paths **cURL**, hitting the OpenAI-compatible chat endpoint directly: ```bash curl -s http://localhost:{port}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` (Substitute the actual `port` captured above — the source's own equivalent example builds this command with an f-string, `f"http://localhost:{port}/v1/chat/completions"`, rather than hardcoding `30000`.) **Python `requests`**, same endpoint: ```python import requests url = f"http://localhost:{port}/v1/chat/completions" data = { "model": "qwen/qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}], } response = requests.post(url, json=data) print(response.json()) ``` **OpenAI Python client**, pointed at the local server: ```python import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[{"role": "user", "content": "List 3 countries and their capitals."}], temperature=0, max_tokens=64, ) print(response) ``` Streaming works the same way with `stream=True`: ```python response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[{"role": "user", "content": "List 3 countries and their capitals."}], temperature=0, max_tokens=64, stream=True, ) for chunk in response: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` **Native `/generate` endpoint**, for lower-level control (parameters detailed in [[concepts/sampling-parameters]]): ```python import requests response = requests.post( f"http://localhost:{port}/generate", json={ "text": "The capital of France is", "sampling_params": {"temperature": 0, "max_new_tokens": 32}, }, ) print(response.json()) ``` Streaming with `/generate` requires manually parsing Server-Sent Events: ```python import requests, json response = requests.post( f"http://localhost:{port}/generate", json={ "text": "The capital of France is", "sampling_params": {"temperature": 0, "max_new_tokens": 32}, "stream": True, }, stream=True, ) prev = 0 for chunk in response.iter_lines(decode_unicode=False): chunk = chunk.decode("utf-8") if chunk and chunk.startswith("data:"): if chunk == "data: [DONE]": break data = json.loads(chunk[5:].strip("\n")) output = data["text"] print(output[prev:], end="", flush=True) prev = len(output) ``` ### 3. Shut down ```python terminate_process(server_process) ``` ### Routing by model type The docs point users toward the API best suited to the model: - Vision language models → OpenAI-compatible vision API (see [[concepts/server-apis]]). - Embedding models → OpenAI-compatible embeddings API, or the native `/encode` endpoint (see [[concepts/server-apis]]). - Reward models → the native `/classify` endpoint (see [[concepts/server-apis]]). ### Skipping the server entirely For batch/local inference without any HTTP server, SGLang's offline `Engine` class can be used directly in-process — see [[concepts/offline-engine]] for the full API (raw/github_doc-docs-docs-basic-usage-offline-engine-api-mdx.md, raw/github_doc-docs-docs-get-started-quickstart-mdx.md). ## Key Parameters - `base_url` — for the OpenAI client, must include the `/v1` suffix (e.g. `http://127.0.0.1:30000/v1`); the server's own routes (e.g. `/generate`) do not. - `api_key` — the cited examples pass the placeholder string `"None"`; the sources don't document general `--api-key`/authentication semantics, so treat "any string works when launched without `--api-key`" as an inference rather than a documented rule. - `stream` — boolean; toggles Server-Sent Events streaming on both the OpenAI-compatible and native `/generate` paths. - `sampling_params` (native `/generate` only) — dict of generation controls; see [[concepts/sampling-parameters]] for the full reference. ## When To Use - Use cURL or plain `requests` for quick manual checks or scripting outside Python's AI ecosystem. - Use the OpenAI Python client when integrating with existing OpenAI-based code, or when you want the richer, higher-level chat/completions interface. - Use the native `/generate` endpoint when you need direct control over low-level sampling parameters, or are calling from a non-OpenAI-shaped client (see [[concepts/server-apis|native API]] material folded into [[concepts/server-apis]]). - Use the offline `Engine` ([[concepts/offline-engine]]) when an HTTP server is unnecessary overhead — e.g., embedding SGLang directly in a batch pipeline or another process. ## Risks & Pitfalls - Forgetting the `/v1` suffix on `base_url` when using the OpenAI client (it appends the specific route, not the version prefix) is likely to cause 404s — the sources show the correct `/v1`-suffixed base URL but don't explicitly document the failure mode of omitting it. - Native `/generate` streaming requires manually parsing the `data: ...` SSE-style lines and detecting the `data: [DONE]` sentinel — this hand-rolled parsing can be easier to get wrong than the OpenAI client's built-in stream iterator, though the sources don't state this comparison directly. - Not waiting for `wait_for_server(...)` (or the `"The server is fired up and ready to roll!"` log line) before sending requests can result in connection failures — a reasonable operator inference from the documented readiness-check pattern, not something the sources state as a guaranteed outcome. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/installation]] - [[concepts/server-apis]] - [[concepts/offline-engine]] - [[concepts/sampling-parameters]] ## Sources - raw/github_doc-docs-docs-basic-usage-send-request-mdx.md - raw/github_doc-docs-docs-basic-usage-offline-engine-api-mdx.md - raw/github_doc-docs-docs-get-started-quickstart-mdx.md --- title: "Server APIs" type: concept tags: [api, basic-usage, foundational, user] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-basic-usage-openai-api-mdx.md", "raw/github_doc-docs-docs-basic-usage-openai-api-completions-mdx.md", "raw/github_doc-docs-docs-basic-usage-openai-api-embeddings-mdx.md", "raw/github_doc-docs-docs-basic-usage-openai-api-vision-mdx.md", "raw/github_doc-docs-docs-basic-usage-native-api-mdx.md", "raw/github_doc-docs-docs-basic-usage-anthropic-api-mdx.md", "raw/github_doc-docs-docs-basic-usage-ollama-api-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition An SGLang server (launched per [[concepts/installation]]) exposes several distinct, simultaneously-available API surfaces on the same process: an **OpenAI-compatible** API (chat completions, completions, embeddings, vision), a **native** SGLang API (`/generate` and a set of operational endpoints), an **Anthropic-compatible** `/v1/messages` API, and an **Ollama-compatible** API. Of these, only the Anthropic-compatible endpoint is explicitly documented as registered automatically on every server with no extra launch flag required (raw/github_doc-docs-docs-basic-usage-anthropic-api-mdx.md). The OpenAI-compatible and native APIs are documented as being "also provided" by the runtime, and the Ollama-compatible API is documented in terms of client-side prerequisites — none of these three sources makes the same explicit "automatic, no-flag, every server" claim the Anthropic doc makes, though in practice all four surfaces are part of the standard server. ## How It Works ### OpenAI-compatible API Covers `chat/completions`, `completions`, embeddings, and vision, enabling a smooth transition from OpenAI services to self-hosted models (raw/github_doc-docs-docs-basic-usage-openai-api-completions-mdx.md). Full parameter references point to the official [OpenAI API Reference](https://platform.openai.com/docs/api-reference). **Chat completions** (`client.chat.completions.create(...)`) auto-applies the Hugging Face tokenizer's chat template (overridable with `--chat-template` at launch): ```python import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[{"role": "user", "content": "List 3 countries and their capitals."}], temperature=0, max_tokens=64, ) ``` It extends the OpenAI schema with `extra_body`, notably `chat_template_kwargs` for model-specific reasoning toggles. Reasoning/thinking output is supported per-model-family, activated by launching with `--reasoning-parser ` and passing the model's own kwarg in `chat_template_kwargs`: | Model family | Chat template parameter | Reasoning parser | Notes | |---|---|---|---| | DeepSeek-R1 (R1, R1-0528, R1-Distill) | `enable_thinking` | `--reasoning-parser deepseek-r1` | Standard reasoning models | | DeepSeek-V3.1 | `thinking` | `--reasoning-parser deepseek-v3` | Hybrid thinking/non-thinking | | Qwen3 (standard) | `enable_thinking` | `--reasoning-parser qwen3` | Hybrid thinking/non-thinking | | Qwen3-Thinking | N/A (always enabled) | `--reasoning-parser qwen3-thinking` | Always generates reasoning | | Kimi | N/A (always enabled) | `--reasoning-parser kimi` | Kimi thinking models | | Gpt-Oss | N/A (always enabled) | `--reasoning-parser gpt-oss` | Gpt-Oss thinking models | Note: SGLang's own documentation is inconsistent about the reasoning-parser name for Kimi models — some upstream docs (including this page's own cited source) use `kimi`, others use `kimi_k2`. The runtime's actual accepted enum (`launch_server --help` / `server_args.py`) is authoritative; verify against your installed version before relying on either spelling. `separate_reasoning: True` (default) returns reasoning in `response.choices[0].message.reasoning_content` separately from `.content`. This behavior is documented specifically for standard Qwen3 models: setting `enable_thinking: False` (or omitting it) yields `reasoning_content = None`, and Qwen3-Thinking models always generate reasoning content regardless of `enable_thinking`. Other model families differ — DeepSeek-V3.1 uses the `thinking` parameter (not `enable_thinking`) instead, and Qwen3-Thinking, Kimi, and Gpt-Oss are documented as always having reasoning enabled with no on/off toggle at all. `logit_bias` is supported on both chat completions and completions: a dict of `{token_id_str: bias}` with bias in `[-100, 100]` (positive boosts, negative suppresses, `-100` effectively bans the token). **MoE expert routing**: setting `return_routed_experts: true` in `extra_body` (requires the server flag `--enable-return-routed-experts`) returns base64-encoded int32 expert IDs shaped `[num_tokens, num_layers, top_k]`. For chat completions this lands in the response-level `sglext` object by default, or per-choice `meta_info` when `return_meta_info: true` is also set. `routed_experts_start_len` in `extra_body` limits the returned range to `[start, seqlen-1)` — useful in multi-turn RL rollouts to avoid re-transferring already-collected routing data. **LoRA adapters**: select an adapter via `model="base-model:adapter-name"` (recommended), or the legacy `extra_body={"lora_path": "adapter_a"}`; `model:adapter` wins if both are given. Server-side, adapters are registered with `--enable-lora --lora-paths adapter_a=/path/to/adapter_a ...`. See [[concepts/lora-and-model-loading]]. **Completions** (`client.completions.create(...)`) is the same idea without `messages`/chat templates — just a `prompt` string: ```python response = client.completions.create( model="qwen/qwen2.5-0.5b-instruct", prompt="List 3 countries and their capitals.", temperature=0, max_tokens=64, n=1, stop=None, ) ``` **Embeddings** (`POST /v1/embeddings` or `client.embeddings.create(...)`): requires launching with `--is-embedding` for decoder-style embedding models; native encoder architectures and `google/embeddinggemma-300m` are auto-detected without the flag. ```python response = client.embeddings.create(model="Alibaba-NLP/gte-Qwen2-1.5B-instruct", input=text) ``` Also accepts `input_ids` directly (tokenized input) instead of raw text, and `encoding_format: "base64"` for compact little-endian FP32 responses when JSON arrays would dominate payload size. **Vision** (`/v1/chat/completions` with multimodal `content` blocks): supports Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3, and more. ```python response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": example_image_url}}, ], }], max_tokens=300, ) ``` Multiple images and interleaved text/images are supported when the model supports it. As an alternative to the server, the offline engine can also run VLM inference directly (see [[concepts/offline-engine]]). ### Native API Beyond the OpenAI-compatible surface, the SGLang runtime exposes its own endpoints (raw/github_doc-docs-docs-basic-usage-native-api-mdx.md): - `/generate` — text generation (similar to `/v1/completions`); full parameters in [[concepts/sampling-parameters]]. - `/get_model_info` — returns `model_path`, `is_generation`, `tokenizer_path`, `preferred_sampling_params`, `weight_version`, `has_image_understanding`, `has_audio_understanding`, `model_type`, `architectures`, and (when applicable) `embedding` (the resolved embedding-serving plan: pooling, normalization, execution/attention style, Matryoshka dimensions, cache policy, effective BCG prefill settings). - `/server_info` — CLI arguments, token limits, memory pool sizes; merges the now-deprecated `get_server_args`, `get_memory_pool_size`, `get_max_total_num_tokens` endpoints. - `/health`, `/health_generate` — liveness checks; the latter generates one token to verify the full generation path. - `/flush_cache` — flushes the radix cache; accepts a `timeout` query param (seconds, default `0` = fail fast if not idle) to wait for idle state, important when HiCache async operations are in flight. Auto-triggered on `/update_weights`. - `/update_weights_from_disk` — hot-swaps model weights from disk (same architecture/size only) without restarting the server, for continuous-eval-during-training workflows; returns `{"success": bool, "message": str}`. - `/encode` — embeddings for embedding models only (errors on generation models); requires launching with `--is-embedding` for decoder-style embedding models (native encoder architectures and `google/embeddinggemma-300m` are auto-detected and don't need the flag). - `/v1/rerank` — cross-encoder document reranking given a query (e.g. `BAAI/bge-reranker-v2-m3`); requires `--attention-backend triton` or `torch_native`, plus `--is-embedding` and typically `--disable-radix-cache --chunked-prefill-size -1`. - `/v1/score` — decoder-only scoring: computes per-item token probabilities for a fixed set of `label_token_ids` given a `query` and `items`; params include `apply_softmax` (default `False`) and `item_first` (default `False`). Useful for classification/response-scoring tasks. - `/classify` — reward-model scoring (SGLang currently treats reward models the same as embedding models); requires `--is-embedding`. - `/start_expert_distribution_record`, `/stop_expert_distribution_record`, `/dump_expert_distribution_record` — capture per-expert selection counts for MoE models, launched with `--expert-distribution-recorder-mode stat`, useful for throughput analysis. - `/tokenize`, `/detokenize` — round-trip tokenization without a model call; `/tokenize` accepts `add_special_tokens`, `/detokenize` accepts `skip_special_tokens`. A full endpoint list lives in `python/sglang/srt/entrypoints/http_server.py` in the source tree. ### Anthropic-compatible API SGLang ships an Anthropic-compatible `POST /v1/messages` endpoint (plus `POST /v1/messages/count_tokens`) so Anthropic SDK clients and agentic CLIs — notably **Claude Code** — can talk to a self-hosted SGLang server unmodified (raw/github_doc-docs-docs-basic-usage-anthropic-api-mdx.md). It reuses the same model, chat template, and reasoning/tool-call parsers as the OpenAI-compatible endpoint, and supports streaming. ```python from anthropic import Anthropic client = Anthropic(base_url="http://127.0.0.1:30000", api_key="EMPTY") message = client.messages.create( model="zai-org/GLM-5.2-FP8", max_tokens=512, messages=[{"role": "user", "content": "List 3 countries and their capitals."}], ) print(next(b.text for b in message.content if b.type == "text")) ``` Note: the Anthropic SDK appends `/v1/messages` itself, so `base_url` is the bare server root (no `/v1`) — unlike the OpenAI client. `system` accepts a string or list of text blocks. Tool definitions follow the Anthropic `tools`/`input_schema` shape; with a `--tool-call-parser` configured, calls come back as `tool_use` content blocks. `POST /v1/messages/count_tokens` tokenizes a request (including system prompt, tools, and history) without generating a response. **Claude Code integration** requires several environment variables: ```bash export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" export ANTHROPIC_AUTH_TOKEN="dummy" export API_TIMEOUT_MS="3000000" export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 export CLAUDE_CODE_ATTRIBUTION_HEADER=0 export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" ``` Critically, `CLAUDE_CODE_ATTRIBUTION_HEADER=0` is required for prefix-cache reuse: Claude Code prepends a per-request attribution block (containing a per-request hash) to the system prompt, which becomes the first token to differ between turns — defeating the radix prefix cache and forcing a full re-prefill of history on every turn. Setting this var removes that attribution line. `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` does **not** cover this — it only silences autoupdater/telemetry/error-reporting traffic, a separate code path. The `[1m]` suffix on the model name is a client-side hint that enables Claude Code's 1M-context beta (SGLang itself does not validate the `model` field on Anthropic-endpoint requests, so any name is accepted; the served model's actual context length is unaffected by the suffix and defaults to the model's native context, e.g. 1,048,576 for GLM-5.2 — use `--context-length` only to cap it). ### Ollama-compatible API Lets the Ollama CLI or Python library treat an SGLang server as the backend, with no Ollama server needed — only the client (`pip install ollama`) (raw/github_doc-docs-docs-basic-usage-ollama-api-mdx.md). | Endpoint | Method | Description | |---|---|---| | `/` | GET, HEAD | Health check for Ollama CLI | | `/api/tags` | GET | List available models | | `/api/chat` | POST | Chat completions (streaming & non-streaming) | | `/api/generate` | POST | Text generation (streaming & non-streaming) | | `/api/show` | POST | Model information | ```bash python -m sglang.launch_server --model Qwen/Qwen2.5-1.5B-Instruct --port 30001 --host 0.0.0.0 ``` ```bash OLLAMA_HOST=http://localhost:30001 ollama list OLLAMA_HOST=http://localhost:30001 ollama run "Qwen/Qwen2.5-1.5B-Instruct" ``` ```python import ollama client = ollama.Client(host='http://localhost:30001') response = client.chat(model='Qwen/Qwen2.5-1.5B-Instruct', messages=[{'role': 'user', 'content': 'Hello!'}]) ``` The model name passed to `ollama run`/the client must exactly match the `--model` value used at launch. A separate "Smart Router" component (documented in the SGLang source tree's `ollama/README.md`, not in this raw set) can route between a fast local Ollama and a more powerful remote SGLang server using an LLM judge. ## Key Parameters - `--chat-template` — override the chat template SGLang auto-detects from the tokenizer (OpenAI-compatible API). - `--reasoning-parser` / `--tool-call-parser` — enable structured reasoning-content and tool-call parsing (shared by the OpenAI and Anthropic endpoints). - `--is-embedding` — required to serve decoder-style embedding/reward models via `/v1/embeddings`, `/encode`, `/classify`. - `--enable-lora` / `--lora-paths` — register named LoRA adapters for the `model:adapter` syntax. - `--enable-return-routed-experts` — required for MoE expert-routing data in responses. - `--attention-backend triton|torch_native` — required for `/v1/rerank` cross-encoder serving. - `CLAUDE_CODE_ATTRIBUTION_HEADER=0` — client-side env var required for prefix-cache reuse when routing Claude Code through SGLang. ## When To Use - **OpenAI-compatible API**: default choice for most integrations — broadest client-library support, chat/completions/embeddings/vision all covered. - **Native API**: when you need operational control (cache flush, weight hot-swap, expert-distribution recording, tokenize/detokenize round-trips) or low-level `/generate` sampling control not exposed by the OpenAI shape. - **Anthropic-compatible API**: to run Claude Code or other Anthropic-SDK-based tooling against a self-hosted model. - **Ollama-compatible API**: to reuse existing Ollama-based tooling/scripts against an SGLang backend, or for local/remote smart-routing setups. ## Risks & Pitfalls - Serving embedding or reward models without `--is-embedding` (for decoder-style architectures) causes `/encode` and `/classify` to error; native encoder architectures and `google/embeddinggemma-300m` are auto-detected and don't need the flag. - `/v1/rerank` has specific backend requirements (`--attention-backend triton` or `torch_native`) — using the default backend will not work for cross-encoder rerank models. - Without a `--tool-call-parser`, tool schemas are still accepted by the Anthropic endpoint, but tool calls return as raw text instead of `tool_use` blocks — this fallback is documented specifically for the Anthropic-compatible API; the OpenAI-compatible sources cited here don't document the identical fallback behavior for `/v1/chat/completions`. - Missing `CLAUDE_CODE_ATTRIBUTION_HEADER=0` when routing Claude Code through SGLang silently degrades performance (full history re-prefill every turn) rather than causing an outright error — easy to miss. - SGLang does not validate the `model` field on Anthropic-endpoint (`/v1/messages`) requests and serves whatever model was loaded at startup; a 404 from `/v1/messages` therefore usually indicates the request didn't reach the route at all (wrong base URL/port), not a model-name mismatch. This is documented specifically for the Anthropic endpoint; no cited OpenAI source makes the same claim for OpenAI-compatible requests. - `flush_cache` with the default `timeout=0` fails fast if the server isn't idle; a non-zero timeout is needed to avoid spurious 400s when HiCache async operations are in-flight (see [[concepts/hierarchical-caching]]). - For Ollama compatibility, the model name in client calls must exactly match the `--model` value at launch. The cited source documents only the exact-match requirement itself — it does not say what happens on a mismatch (silent failure, error, or otherwise), so don't assume a specific failure mode. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/sending-requests]] - [[concepts/offline-engine]] - [[concepts/sampling-parameters]] - [[concepts/lora-and-model-loading]] - [[concepts/structured-outputs-and-tool-calling]] - [[concepts/hierarchical-caching]] - [[concepts/router-and-model-gateway]] - [[concepts/speculative-decoding]] ## Sources - raw/github_doc-docs-docs-basic-usage-openai-api-mdx.md - raw/github_doc-docs-docs-basic-usage-openai-api-completions-mdx.md - raw/github_doc-docs-docs-basic-usage-openai-api-embeddings-mdx.md - raw/github_doc-docs-docs-basic-usage-openai-api-vision-mdx.md - raw/github_doc-docs-docs-basic-usage-native-api-mdx.md - raw/github_doc-docs-docs-basic-usage-anthropic-api-mdx.md - raw/github_doc-docs-docs-basic-usage-ollama-api-mdx.md --- title: "Server Arguments" type: concept tags: [api, architecture, parallelism, caching, quantization, operator, foundational, well-established] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md", "raw/github_doc-docs-cookbook-base-reference-server-arguments-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition `python3 -m sglang.launch_server` is configured entirely through CLI flags (or an equivalent `--config config.yaml`). The full reference (`raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md`) is organized into **39 sections** covering well over 200 flags — from model/tokenizer selection through PD disaggregation, MindStudio-probe dumps, and deprecated arguments. This page groups the operationally important flags by area with verbatim names and defaults; it is not a full transcription. The complete, current list is always `python3 -m sglang.launch_server --help` or `python/sglang/srt/server_args.py` in the source repo. ## How It Works ### Common launch patterns ```bash Command # Config file (CLI flags override it) python -m sglang.launch_server --config config.yaml # Tensor parallelism across 2 GPUs python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 2 # Data parallelism (recommended via the router/gateway rather than --dp directly) python -m sglang_router.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --dp 2 --tp 2 # Reduce KV cache memory footprint on OOM python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --mem-fraction-static 0.7 # Reduce chunked prefill size on prefill-time OOM for long prompts python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --chunked-prefill-size 4096 # Multi-node TP=4 across 2 nodes x 2 GPUs python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 \ --dist-init-addr sgl-dev-0:50000 --nnodes 2 --node-rank 0 # (repeat with --node-rank 1 on node 1) ``` (raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md) Other one-liners called out in the source: fp8 weight quant via `--quantization fp8`; fp8 KV cache via `--kv-cache-dtype fp8_e4m3` or `fp8_e5m2`; deterministic inference via `--enable-deterministic-inference`; restrict multimodal remote-media fetches to a domain allowlist via `--allowed-media-domains` (default: any domain allowed, for backward compatibility — don't expose that default to untrusted clients); decode context parallelism via `--dcp-size N`; custom chat templates via `--hf-chat-template-name` when the tokenizer has multiple named templates. ### Model and tokenizer | Flag | Default | Notes | |---|---|---| | `--model-path` / `--model` | `None` | Local folder or HF repo ID | | `--tokenizer-path` | `None` | Defaults to the model path | | `--load-format` | `auto` | `auto`, `pt`, `safetensors`, `gguf`, `bitsandbytes`, `layered` (quantize-as-you-load), `flash_rl`, `remote_instance` (R-Fork), `runai_streamer`, etc. | | `--trust-remote-code` | `False` | Allow custom modeling code from the Hub | | `--context-length` | `None` | Overrides `config.json`'s max context | | `--is-embedding` | `False` | Use a CausalLM as an embedding model | | `--enable-multimodal` | `None` | Enable multimodal handling for a multimodal model | | `--revision` | `None` | Branch/tag/commit pin | ### HTTP server | Flag | Default | Notes | |---|---|---| | `--host` | `127.0.0.1` | | | `--port` | `30000` | | | `--grpc-mode` | `False` | Use gRPC instead of HTTP | | `--ssl-keyfile` / `--ssl-certfile` / `--ssl-ca-certs` / `--ssl-keyfile-password` | `None` | TLS termination | | `--enable-ssl-refresh` | `False` | Hot-reload certs on change | | `--enable-http2` | `False` | Granian ASGI server instead of Uvicorn; needs `pip install sglang[http2]` | ### Quantization and data type | Flag | Default | Notes | |---|---|---| | `--dtype` | `auto` | `auto`, `half`/`float16`, `bfloat16`, `float`/`float32` | | `--quantization` | `None` | `awq`, `fp8`, `gptq`, `marlin`, `bitsandbytes`, `gguf`, `modelopt`/`modelopt_fp8`/`modelopt_fp4`, `w8a8_int8`, `w8a8_fp8`, `mxfp4`, `compressed-tensors`, and more | | `--kv-cache-dtype` | `auto` | `fp8_e5m2`, `fp8_e4m3` (CUDA 11.8+), `nvfp4`/`fp4_mx_block16` (CUDA 12.8+, PyTorch 2.8+) | | `--quantization-param-path` | `None` | KV-cache scaling factors JSON; should be set when `--kv-cache-dtype` is fp8 to avoid accuracy loss | | `--enable-fp32-lm-head` | `False` | FP32 logits | A separate cookbook quick-reference (`raw/github_doc-docs-cookbook-base-reference-server-arguments-mdx.md`) maps parallelism config fields to CLI flags: `tp` → `--tp-size`/`--tensor-parallel-size`; `dp` → `--dp-size`/`--data-parallel-size`; `ep` → `--ep-size`/`--expert-parallel-size`/`--ep`; `enable_dp_attention` → `--enable-dp-attention` (DP for attention, TP for FFN — a hybrid mode). ### Memory and scheduling | Flag | Default | Notes | |---|---|---| | `--mem-fraction-static` | `None` (auto-computed when unset; falls back to `0.88` only if GPU memory can't be detected) | `(model weights + KV cache pool) / GPU memory`; when unset, computed as `(GPU memory - reserved memory) / GPU memory`; lower on OOM | | `--max-running-requests` | `None` | Concurrency cap | | `--max-total-tokens` | `None` | KV pool token cap (mainly debug use) | | `--chunked-prefill-size` | `None` | `-1` disables chunked prefill | | `--max-prefill-tokens` | `16384` | Real bound is `max(this, model max context)` | | `--schedule-policy` | `fcfs` | `lpm` (longest-prefix-match), `random`, `dfs-weight`, `lof`, `priority`, `routing-key` | | `--schedule-conservativeness` | `1.0` | Raise if requests are frequently retracted | | `--page-size` | `1` | Tokens per KV cache page | | `--radix-eviction-policy` | `lru` | `lru`, `lfu`, `slru`, `priority` | | `--enable-priority-scheduling` / `--priority-scheduling-preemption-threshold` | `False` / `10` | Higher-priority requests can preempt | See `[[concepts/architecture-and-radixattention]]` for how these interact with the scheduler and RadixAttention. ### Parallelism (tensor / pipeline / data / expert / context) | Flag | Default | Notes | |---|---|---| | `--tensor-parallel-size` / `--tp-size` | `1` | | | `--pipeline-parallel-size` / `--pp-size` | `1` | | | `--data-parallel-size` / `--dp-size` | `1` | Better for throughput when memory allows; SGLang docs recommend the Model Gateway/router over raw `--dp-size` — see `[[concepts/router-and-model-gateway]]` | | `--expert-parallel-size` / `--ep-size` / `--ep` | `1` | MoE expert distribution | | `--moe-data-parallel-size` / `--moe-dp-size` | `1` | | | `--dcp-size` / `--decode-context-parallel-size` | `1` | Decode context parallelism for MLA models | | `--attention-context-parallel-size` / `--attn-cp-size` | `1` | | | `--load-balance-method` | `auto` | DP load balancing; `total_tokens` needs DP attention | ### Multi-node distributed serving | Flag | Default | Notes | |---|---|---| | `--dist-init-addr` / `--nccl-init-addr` | `None` | e.g. `192.168.0.2:25000` | | `--nnodes` | `1` | | | `--node-rank` | `0` | | ### API, chat templates, and parsers | Flag | Default | Notes | |---|---|---| | `--api-key` | `None` | Bearer auth for the OpenAI-compatible server | | `--admin-api-key` | `None` | Separate key gating admin endpoints (weights update, cache flush, `/server_info`) | | `--served-model-name` | `None` | Override `/v1/models` name | | `--chat-template` / `--hf-chat-template-name` | `None` | Built-in name, file path, or named-template selection when the tokenizer has several | | `--reasoning-parser` | `None` | `deepseek-r1`, `deepseek-v3`, `glm45`, `gpt-oss`, `kimi`, `qwen3`, `qwen3-thinking`, `step3` | | `--tool-call-parser` | `None` | `deepseekv3`, `glm45`, `gpt-oss`, `kimi_k2`, `llama3`, `mistral`, `pythonic`, `qwen25`, `qwen3_coder`, `step3`, `gigachat3`, etc. | | `--sampling-defaults` | `model` | `model` (use `generation_config.json`) or `openai` (temperature=1.0, top_p=1.0, ...) | Note: SGLang's own documentation is inconsistent about the Kimi reasoning-parser value — this page's cited source spells it `kimi` for `--reasoning-parser` (note the *tool-call* parser is separately and consistently spelled `kimi_k2`); other upstream docs use `kimi_k2` for the reasoning parser too. The runtime's actual accepted enum (`launch_server --help` / `server_args.py`) is authoritative; verify against your installed version before relying on either spelling. ### LoRA | Flag | Default | Notes | |---|---|---| | `--enable-lora` | `False` | Auto-set if `--lora-paths` is given | | `--lora-paths` | `None` | `` \| `=` \| JSON `{"lora_name", "lora_path", "pinned"}` | | `--max-loras-per-batch` | `8` | Adapters per running batch, including base-only requests | | `--max-lora-rank` | `None` | Inferred from adapters unless dynamically loading larger ones later | | `--max-loaded-loras` | `None` | CPU-resident cap; must be ≥ `--max-loras-per-batch` | | `--lora-eviction-policy` | `lru` | `lru`, `fifo` | | `--lora-backend` | `csgmv` | `triton`, `csgmv`, `ascend`, `torch_native` | | `--enable-lora-overlap-loading` | `False` | Overlap adapter H2D transfer with compute | ### Kernel backends (attention, sampling, grammar, GEMM) | Flag | Default | Notes | |---|---|---| | `--attention-backend` | `None` | `triton`, `fa3`, `fa4`, `flashinfer`, `flashmla`, `trtllm_mla`, `cutlass_mla`, `aiter`, `ascend`, `intel_amx`, `intel_xpu`, and more | | `--prefill-attention-backend` / `--decode-attention-backend` | `None` | Override `--attention-backend` per phase | | `--sampling-backend` | `None` | `flashinfer`, `pytorch`, `ascend` | | `--grammar-backend` | `None` | `xgrammar`, `outlines`, `llguidance`, `none` | | `--mm-attention-backend` | `None` | Multimodal-specific attention backend | | `--fp8-gemm-backend` / `--fp4-gemm-backend` / `--bf16-gemm-backend` | `auto` | Hardware-dependent GEMM kernel selection (DeepGEMM, FlashInfer CUTLASS/CuTe-DSL/TRT-LLM, Marlin, cuBLAS, ...) | ### Speculative decoding | Flag | Default | Notes | |---|---|---| | `--speculative-algorithm` | `None` | `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`, `DFLASH` | | `--speculative-draft-model-path` | `None` | Local folder or HF repo ID | | `--speculative-num-steps` | `None` | Draft-model sampling steps | | `--speculative-eagle-topk` | `None` | Tokens sampled per EAGLE2 step | | `--speculative-num-draft-tokens` | `None` | Tokens sampled per Speculative Decoding pass | | `--speculative-accept-threshold-single` | `1.0` | Accept a draft token if its target-model probability exceeds this threshold | | `--speculative-accept-threshold-acc` | `1.0` | Raises a draft token's accept probability from its target probability `p` to `min(1, p / threshold_acc)` | Note: the source's own summary enum for `--speculative-algorithm` (reproduced above) omits `DFLASH`, but the same raw file's detailed flags — `--speculative-dflash-block-size`, `--speculative-draft-window-size` (DFLASH-specific behavior), `--speculative-dflash-draft-window-size` — confirm `DFLASH` is a real accepted value. This is an internal inconsistency in the raw source itself, not a wiki error; treat the runtime's actual enum (`launch_server --help` / `server_args.py`) as authoritative. Ngram and multi-layer Eagle variants have their own dedicated sections in the source with additional flags not reproduced here. ### MoE / expert parallelism | Flag | Default | Notes | |---|---|---| | `--expert-parallel-size` / `--ep-size` / `--ep` | `1` | | | `--moe-a2a-backend` | `none` | `deepep`, `mooncake`, `nixl`, `mori`, `flashinfer`, `pplx`, etc. — all-to-all comm for EP | | `--moe-runner-backend` | `auto` | `deep_gemm`, `triton`, `flashinfer_trtllm`, `cutlass`, `aiter`, `marlin`, etc. | | `--deepep-mode` | `auto` | `normal` (prefill), `low_latency` (decode), or `auto` | | `--enable-eplb` / `--eplb-algorithm` | `False` / `auto` | Expert-parallel load balancing | | `--ep-num-redundant-experts` | `0` | Redundant expert replicas for hot experts | | `--enable-waterfill` | `False` | Route the fused shared expert to the least-loaded EP rank; supports the DeepEP and MegaMOE all-to-all backends; supported on DeepSeek-V3/R1 models with EP >= 2 | ### PD disaggregation | Flag | Default | Notes | |---|---|---| | `--disaggregation-mode` | `null` | `prefill` or `decode`; unset = not disaggregated | | `--disaggregation-transfer-backend` | `mooncake` | `mooncake`, `nixl`, `ascend`, `mori`, `mooncake_tcp`, `fake` | | `--disaggregation-bootstrap-port` | `8998` | On the prefill server | | `--disaggregation-decode-enable-radix-cache` | `False` | Cache KV prefixes on the decode server to avoid redundant transfers; incompatible with HiSparse, speculative decoding, and the `fake` transfer backend | | `--num-reserved-decode-tokens` | `512` | Reserved per-request decode-token memory | ### Hierarchical cache (HiCache) Covered in depth in `[[concepts/architecture-and-radixattention]]`. Key flags: `--enable-hierarchical-cache`, `--hicache-ratio` / `--hicache-size`, `--page-size`, `--hicache-storage-prefetch-policy`, `--hicache-write-policy`, `--hicache-io-backend`, `--hicache-mem-layout`, `--hicache-storage-backend` (`file`, `mooncake`, `hf3fs`, `nixl`, `aibrix`, `dynamic`, `eic`, `simm`). ## Key Parameters Curator/operator guidance, not a ranking sourced from the docs: the source's own recommendations are narrower — it specifically calls out lowering `--mem-fraction-static` on OOM and raising `--schedule-conservativeness` if requests are frequently retracted, and points to a separate hyperparameter-tuning guide for the rest rather than ranking a fixed group of flags as universally "first to tune." With that caveat, **memory and scheduling** (`--mem-fraction-static`, `--chunked-prefill-size`, `--schedule-conservativeness`, `--max-running-requests`) is a reasonable starting group for OOM or throughput issues — see `[[concepts/architecture-and-radixattention]]`. For multi-GPU/multi-node, `--tp-size`/`--dp-size`/`--ep-size` and `--nnodes`/`--dist-init-addr` are load-bearing. ## When To Use Every SGLang deployment goes through these flags — they're not optional advanced configuration, they *are* the configuration surface (`launch_server` has no separate "simple mode"). Start from the "Common launch patterns" above and layer in a section (quantization, LoRA, speculative decoding, PD, MoE/EP) only as the workload requires it. ## Risks & Pitfalls - Many flags interact: e.g. `--disaggregation-decode-enable-radix-cache` is explicitly incompatible with HiSparse, speculative decoding, and the `fake` transfer backend; `--enable-unified-memory` requires specific attention backends and is incompatible with PD disaggregation and speculative decoding (raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md). - `--allowed-media-domains` defaults to allowing any HTTP(S) domain for multimodal media fetches "for backward compatibility" — don't rely on the default in an untrusted-client deployment. - Several flags are marked deprecated in the source (`--stream-output`, `--nsa-prefill-backend`/`--nsa-decode-backend` → use `--dsa-*` instead) — verbatim names are given here as documented. Deprecations in the source are actually distributed throughout the reference (marked inline, per-flag, in their own sections — e.g. `--stream-output` under Runtime options, the `--nsa-*` flags under Kernel backends) rather than centralized: the dedicated "Deprecated arguments" section only lists `--prefill-round-robin-balance` and `--hybrid-kvcache-ratio`, so don't rely on that section alone for the current list. - `torch.compile` (`--enable-torch-compile`) is noted in the source as "out of maintenance and might cause error." ## Related Concepts - `[[concepts/architecture-and-radixattention]]` — scheduler and RadixAttention mechanics behind the memory/scheduling flags - `[[concepts/router-and-model-gateway]]` — router-side flags (`--policy`, `--pd-disaggregation`, etc.) are documented separately from `launch_server` flags - `[[concepts/frontend-dsl]]` — client-side generation parameters, distinct from these server-launch flags - `[[concepts/parallelism-and-disaggregation]]` — deeper TP/PP/DP/EP/PD treatment - `[[concepts/lora-and-model-loading]]` — deeper LoRA treatment - `[[concepts/attention-backends-and-cuda-graph]]` — deeper kernel-backend treatment - `[[concepts/speculative-decoding]]` — deeper EAGLE/NGRAM treatment - `[[concepts/quantization]]` — deeper quant format treatment ## Sources - raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md - raw/github_doc-docs-cookbook-base-reference-server-arguments-mdx.md --- title: "SGLang Overview" type: concept tags: [overview, foundational, user] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-readme-md.md", "raw/github_doc-docs-docs-get-started-quickstart-mdx.md", "raw/github_doc-docs-docs-basic-usage-overview-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition SGLang is a high-performance serving framework for large language models (LLMs) and multimodal models. It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters. SGLang is open-source, hosted under the non-profit organization LMSYS. The upstream README itself claims SGLang "has become the de facto industry standard" for open-source LLM inference and reports deployments running on over 400,000 GPUs worldwide (raw/github_doc-readme-md.md, 2026-08-24 snapshot) — these are the project's own self-reported claims, not independently verified by this KB. ## How It Works This KB frames SGLang's position as resting on three pillars — this is curator synthesis for organizing this wiki, not a "three pillars" architecture that the basic-usage overview doc itself states; that source is simply a flat list of links to the OpenAI/Anthropic/Ollama/offline-engine/native/sampling-parameters/model-usage docs (raw/github_doc-docs-docs-basic-usage-overview-mdx.md): 1. **A fast runtime.** The core engine provides efficient serving through RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, quantization (FP4/FP8/INT4/AWQ/GPTQ), and multi-LoRA batching (raw/github_doc-readme-md.md). See [[concepts/architecture-and-radixattention]] for the caching mechanics. 2. **A frontend DSL.** SGLang also ships a frontend DSL (see [[concepts/frontend-dsl]]); none of this page's three cited sources defines or substantively describes it (the README only links a "Frontend Tutorial" without elaborating), so treat any definition of it here as editorial framing rather than a sourced claim. 3. **An OpenAI-compatible server.** A served HTTP API surface — OpenAI-compatible endpoints, plus native, Anthropic-compatible, and Ollama-compatible APIs — so existing client tooling can talk to a self-hosted SGLang deployment. The "without changes" / drop-in claim is explicitly documented for the OpenAI-compatible API only (raw/github_doc-docs-docs-get-started-quickstart-mdx.md: "SGLang is fully OpenAI API-compatible"); the overview doc lists the Anthropic- and Ollama-compatible docs but does not itself state drop-in/no-changes behavior for them. See [[concepts/server-apis]] for the full API surface and [[concepts/offline-engine]] for in-process usage without a server. A typical usage flow (raw/github_doc-docs-docs-get-started-quickstart-mdx.md): install the package, launch an inference server via `python3 -m sglang.launch_server`, then send requests using cURL, the OpenAI Python client, plain `requests`, or SGLang's native `/generate` endpoint. See [[concepts/installation]] and [[concepts/sending-requests]] for the mechanics of each step. ## Key Parameters - **Broad model support** — language models (Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, Mistral, etc.), embedding models (e5-mistral, gte, mcdse), reward models (Skywork), and diffusion models (WAN, Qwen-Image); compatible with most Hugging Face models and OpenAI APIs (raw/github_doc-readme-md.md). See [[concepts/supported-models]]. - **Extensive hardware support** — NVIDIA GPUs (GB200/B300/H100/A100/Spark/5090), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, and more (raw/github_doc-readme-md.md). See [[concepts/supported-hardware]]. - **RL & post-training backbone** — used as a rollout backend for training frontier models, with native RL integrations and adoption by post-training frameworks such as AReaL, Miles, slime, Tunix, and verl (raw/github_doc-readme-md.md). ## When To Use SGLang targets teams that need to self-host LLM/multimodal inference at low latency and high throughput, whether on a single GPU or a large distributed cluster, and that want drop-in, no-changes compatibility with OpenAI-style client code (raw/github_doc-docs-docs-get-started-quickstart-mdx.md) rather than building request/response plumbing from scratch. SGLang also ships Anthropic- and Ollama-compatible APIs (raw/github_doc-docs-docs-basic-usage-overview-mdx.md), but the "no changes" / drop-in framing is specifically documented for OpenAI compatibility — see [[concepts/server-apis]] for what's actually documented per API surface. It is also positioned as a production-grade rollout backend for RL/post-training pipelines (raw/github_doc-readme-md.md). ## Risks & Pitfalls - The README's "News" section and feature list describe a fast-moving project (frequent day-0 model support announcements, active blog cadence); pinning a specific version (this KB targets v0.5.18) matters for reproducibility. Note that the underlying raw docs cited throughout this KB are mutable `main`-branch snapshots fetched 2026-08-24, not files pinned to the v0.5.18 tag — see [[concepts/installation]] for version-pinning guidance with Docker and source installs. - SGLang layers several serving APIs (OpenAI, native, Anthropic, Ollama) on one server; picking the wrong one for a task (e.g., using the native `/generate` endpoint when an OpenAI-compatible client library is expected) adds unnecessary complexity — see [[concepts/server-apis]]. ## Related Concepts - [[concepts/installation]] - [[concepts/sending-requests]] - [[concepts/server-apis]] - [[concepts/offline-engine]] - [[concepts/sampling-parameters]] - [[concepts/architecture-and-radixattention]] - [[concepts/frontend-dsl]] - [[concepts/supported-models]] - [[concepts/supported-hardware]] ## Sources - raw/github_doc-readme-md.md - raw/github_doc-docs-docs-get-started-quickstart-mdx.md - raw/github_doc-docs-docs-basic-usage-overview-mdx.md --- title: "Speculative Decoding" type: concept tags: [advanced, well-established, api, operator] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-speculative-decoding-mdx.md" - "raw/github_doc-docs-docs-advanced-features-adaptive-speculative-decoding-md.md" - "raw/github_doc-docs-docs-advanced-features-dcp-mdx.md" - "raw/github_doc-docs-docs-advanced-features-hisparse-guide-mdx.md" confidence: medium --- # Speculative Decoding ## Definition Speculative decoding accelerates autoregressive generation by having a cheap "draft" mechanism propose several candidate future tokens per step, which the full target model then verifies (accepts or rejects) in a single parallel forward pass — turning many sequential single-token decode steps into fewer, larger verification steps. SGLang implements several draft mechanisms (EAGLE-2, EAGLE-3, MTP, DFLASH, STANDALONE, NGRAM) plus **adaptive speculative decoding**, which retunes the draft depth at runtime as acceptance behavior changes. As documented in the SGLang docs, verified against `main` fetched 2026-08-24 — the cited sources are mutable `main`-branch snapshots, not files pinned to the `v0.5.18` tag, so treat parameter defaults/tables below as accurate as of that fetch date rather than guaranteed-stable for the tag. Note: `server-arguments.md` (edited separately) may omit DFLASH from its condensed algorithm-enum table due to an upstream source inconsistency — trust this page's DFLASH coverage instead. ## How It Works ### Method landscape | Method | Draft source | Separate draft model? | Enable | Constraints | |---|---|---|---|---| | EAGLE-2 | EAGLE draft model (feature drafting + tree) | Typically yes | `--speculative-algorithm EAGLE` + `--speculative-draft-model-path ...` | Tune `--speculative-num-steps/-eagle-topk/-num-draft-tokens` | | EAGLE-2 + FR-Spec | Same + truncated high-freq token vocab | Typically yes | add `--speculative-token-map ...` | Reduces `lm_head` overhead; ignored for EAGLE3 | | EAGLE-3 | EAGLE3 draft model | Yes | `--speculative-algorithm EAGLE3` + `--speculative-draft-model-path ...` | Best throughput in the benchmark below (which compares no-speculation, EAGLE-2, and EAGLE-3 only — not MTP/DFLASH/STANDALONE/NGRAM) | | MTP | Built-in multi-token heads (model-specific) | Often no | via speculative workflow, small step/topk/draft-token counts | Draft path may be auto-handled | | DFLASH | DFlash draft model (linear block verification) | Yes | `--speculative-algorithm DFLASH` + `--speculative-draft-model-path ...` | No `--enable-dp-attention`; `pp_size==1`; disables overlap scheduler and mixed chunked prefill | | STANDALONE | Smaller draft LLM (token-level) | Yes | `--speculative-algorithm STANDALONE` + `--speculative-draft-model-path ...` | No `--enable-dp-attention` | | NGRAM | Ngram cache from prior tokens | No | `--speculative-algorithm NGRAM` | CUDA-only; no `--enable-dp-attention`; disables overlap scheduler and mixed chunked prefill | `--speculative-algorithm NEXTN` is an alias of `EAGLE`. Quick guidance from the docs: EAGLE-3 for best speed/quality; EAGLE-2 for a strong, broadly compatible default; add adaptive speculative decoding on top of EAGLE with `--speculative-eagle-topk 1` when acceptance varies over time; FR-Spec (`--speculative-token-map`) to cut `lm_head` overhead on EAGLE-2; MTP when the model has built-in multi-token heads; DFLASH when a DFlash draft checkpoint exists; STANDALONE with a smaller draft LLM; NGRAM when no extra model is available at all (CUDA-only). Upstream-reported benchmark (2026-08-24 snapshot; LLaMA-3.1-8B-Instruct, MT-Bench, 1×H100): no speculative decoding 158.34 tok/s → EAGLE-2 244.10 tok/s → EAGLE-3 373.25 tok/s. ### EAGLE mechanics The EAGLE draft model predicts the next *feature vector* (the target LLM's last hidden state), using the feature sequence and preceding tokens; the next token is sampled from `LMHead(feature)`, then the two sequences extend in a **tree**, branching by `--speculative-eagle-topk` per step. SGLang's EAGLE-2 expands the draft tree for `--speculative-num-steps` steps, then reranks to select the top `--speculative-num-draft-tokens` final nodes as the actual draft tokens sent for verification. EAGLE-3 drops the feature-prediction objective, incorporates low/mid-layer features, and trains on-policy — this is why it out-throughputs EAGLE-2. Train custom EAGLE-3 drafters with SGLang's [SpecForge](https://github.com/sgl-project/SpecForge) framework. Core EAGLE parameters: | Parameter | Meaning | Default | |---|---|---| | `--speculative-draft-model-path` | Draft model weights (required for EAGLE/EAGLE3/STANDALONE; optional for some MTP models) | `None` | | `--speculative-num-steps` | Autoregressive drafting depth; deeper = more speculation range but more rejection-cascade risk | Auto (5 for Llama/Grok, 3 for most others) | | `--speculative-eagle-topk` | Branching factor per step; higher = more diversity/acceptance but more memory/compute | Auto (4 for Llama/Grok, 1 for most others) | | `--speculative-num-draft-tokens` | Max parallel verification capacity | Auto (8 for Llama/Grok, 4 for most others; forced to `num_steps+1` if `topk=1`) | | `--speculative-accept-threshold-single` / `--speculative-accept-threshold-acc` | Single-token / accumulated acceptance thresholds (lower = more aggressive acceptance) | `1.0` / `1.0` | | `--speculative-attention-mode` | `prefill` or `decode`, affects target verification and draft extension | `"prefill"` | | `--speculative-draft-attention-backend` | Override attention backend for the draft model | `None` (same as target) | | `--speculative-draft-model-quantization` | Draft-model quantization override; `"unquant"` forces no quantization even if target is quantized | same as target | | `--speculative-moe-runner-backend`, `--speculative-moe-a2a-backend` | Draft-model-specific MoE backend overrides | `None` | Leave `num-steps`/`eagle-topk`/`num-draft-tokens` all unset for auto-tuning, or set all three explicitly — use `scripts/playground/bench_speculative.py` to find the best combination. Example (EAGLE-2): ```bash python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-num-steps 3 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 --cuda-graph-max-bs-decode 8 --log-level warning ``` Add `--enable-torch-compile` (optionally `--torch-compile-max-bs`) for kernel-level optimization on the draft model — benefit is hardware/model-dependent and should be benchmarked. FR-Spec adds `--speculative-token-map ` (e.g. from `thunlp/LLaMA3-Instruct-8B-FR-Spec`) to shrink `lm_head` cost via a truncated high-frequency vocabulary. EAGLE-3 example just swaps `--speculative-algorithm EAGLE3` and a matching EAGLE3 draft checkpoint. ### MTP (Multi-Token Prediction) Models with built-in multi-token-prediction heads (e.g. `XiaomiMiMo/MiMo-7B-RL`, DeepSeek-V3.2) use the EAGLE speculative pathway with very small step counts, e.g. `--speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`. ### DFLASH Verifies a **linear draft block** (not a tree) using a dedicated DFlash draft checkpoint. Parameters: `--speculative-draft-model-path` (required), `--speculative-num-draft-tokens` (verify block size, inferred from draft config or defaults to 16), `--speculative-dflash-block-size` (alias of the above for DFlash), `--speculative-dflash-draft-window-size` (draft KV sliding-window size, must be ≥ `speculative-num-draft-tokens` when set). Constraints: no `--enable-dp-attention`, `pp_size == 1`, disables the overlap scheduler and mixed chunked prefill. ```bash python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct \ --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat ``` ### STANDALONE (small draft model) Classic token-level speculative decoding with a smaller separate LLM as the draft model. Auto defaults: `--speculative-num-steps 3`, `--speculative-eagle-topk 1`, `--speculative-num-draft-tokens 4`. Does not support `--enable-dp-attention`. ```bash python3 -m sglang.launch_server --model Qwen/Qwen2.5-7B-Instruct \ --speculative-algorithm STANDALONE --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \ --speculative-num-steps 4 --speculative-eagle-topk 2 --speculative-num-draft-tokens 7 \ --mem-fraction-static 0.7 --cuda-graph-max-bs-decode 8 --log-level warning ``` ### Speculative Decoding V2 (overlap scheduler) V2 speculative workers (`StandaloneWorkerV2`, `EAGLEWorkerV2`) run with the overlap scheduler enabled **by default**; disable with `--disable-overlap-schedule` to fall back to synchronous execution. The overlap scheduler only supports `--speculative-eagle-topk 1` — explicitly setting `topk > 1` errors, and *omitting* topk risks auto-tuning silently picking `topk > 1` for some models (e.g. Llama) without an immediate config error, so **always set `--speculative-eagle-topk 1` explicitly** when relying on the overlap scheduler. ### NGRAM (no draft model) Retrieves draft tokens from an ngram cache built from previously generated tokens, then verifies with the target model — no extra model needed. | Parameter | Default | Meaning | |---|---|---| | `--speculative-num-draft-tokens` | `12` (or `min(--speculative-ngram-max-trie-depth, 12)` if unset) | Draft tokens verified per step | | `--speculative-ngram-min-bfs-breadth` / `-max-bfs-breadth` | `1` / `10` | BFS breadth bounds | | `--speculative-ngram-match-type` | `"BFS"` | `"BFS"` (recency-based) or `"PROB"` (frequency-based) tree building | | `--speculative-ngram-max-trie-depth` | `18` | Max suffix length stored/matched | | `--speculative-ngram-capacity` | `10,000,000` | Cache entry capacity | Constraints: CUDA-only, no `--enable-dp-attention`, disables overlap scheduler and mixed chunked prefill; if `--speculative-ngram-max-bfs-breadth > 1` (i.e. effective `eagle_topk > 1`) **and** `page_size > 1`, you must set `--attention-backend flashinfer` or the server errors. `SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True` forces greedy verification. ### Other flags `--enable-multi-layer-eagle` (auto-enabled for MiMoV2 and Step3p5 models). ### Adaptive speculative decoding Retunes `speculative_num_steps`/`speculative_num_draft_tokens` **at runtime** instead of a single fixed value, because the optimal step count depends on current acceptance rate and batch size (too few steps wastes acceptable draft capacity; too many wastes compute on tokens the target model will reject — and at high batch size, each wasted draft step is multiplied across every sequence in the batch). **Currently only supported** for `--speculative-algorithm EAGLE`/`EAGLE3` with `--speculative-eagle-topk 1`; otherwise SGLang silently falls back to static settings. Architecture: `AdaptiveSpeculativeParams` (EMA-based policy) + `SpecRuntimeState` (per-tier runtime state: attention backend + CUDA graph for draft/verify/extend stages) + `AdaptiveController` (queries the policy per current batch size, activates the matching state). The controller keeps **independent EMA trackers per batch-size range** (BS ranges defined as lower-bound keys in the config, e.g. `"1"` and `"8"` mean BS 1–7 vs. BS 8+), so small-BS acceptance signal doesn't pollute large-BS decisions. Tier switching is a reference swap between pre-captured CUDA graphs, never an online recapture, and only happens after the current speculative round completes. Runtime flow per decode step: (0) `activate_step_by_batch(batch_size)` — query optimal step for current BS, activate if different → (1) `draft()` → (2) `verify()` (produces accepted-length per request) → (3) `forward_draft_extend_after_decode()` (draft KV catch-up) → (4) `adaptive_controller.on_verify_complete(...)` updates the EMA for the matching BS slot, applying warmup/interval/hysteresis gates, and selects a new tier if warranted. Decision rule (conceptual): `target_steps ≈ clamp(round(ema_accept_len) + 1, min(candidate_steps), max(candidate_steps))` — consistently high acceptance nudges the tier up, early rejection nudges it down. Guard rails: `warmup_batches` (skip initial batches), `update_interval` (avoid switching every batch), `up_hysteresis`/`down_hysteresis` (reduce oscillation), `ceiling_coeff` (optional — caps `num_steps` proportional to observed draft quality at high BS). Enable: ```bash python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-eagle-topk 1 --speculative-num-steps 3 --speculative-num-draft-tokens 4 \ --speculative-adaptive ``` Override defaults with `--speculative-adaptive-config /path/to/adaptive_spec.json`: ```json { "ema_alpha": 0.2, "warmup_batches": 10, "update_interval": 5, "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0}, "8": {"candidate_steps": [1], "up_hysteresis": 0.0, "down_hysteresis": 0.0, "ceiling_coeff": 0} } ``` Non-integer keys (`ema_alpha`, `warmup_batches`, `update_interval`) are global; integer keys are per-BS slots and each must specify `candidate_steps` (non-empty list of positive ints) or the config errors. Built-in conservative default (BS 1: `[1,3,7]`, BS 8–31: `[1,3]`, BS≥32: `[1]`) suits weak draft models (MiniMax-M2.5, DSV4); an aggressive preset with `ceiling_coeff > 0` suits strong/high-variance drafters (GLM-4.7-FP8). Monitor via `curl -s http://127.0.0.1:30000/server_info | jq '.internal_states[0] | {speculative_num_steps, avg_spec_accept_length}'`. ### OOM troubleshooting Speculative decoding raises VRAM use (draft tree + CUDA graphs + verification buffers). Remediation order: (1) lower `--mem-fraction-static` (e.g. `0.5`) — most effective single lever; (2) reduce `--cuda-graph-max-bs-decode` (e.g. `4` or `2`); (3) shrink the draft tree (`--speculative-num-steps`, `--speculative-eagle-topk`, `--speculative-num-draft-tokens` — e.g. from `5/8/64` down to `3/1/4`); (4) limit `--max-running-requests`. Quick recovery recipe combines all four at conservative values, then scale `num-draft-tokens`/`eagle-topk`/`cuda-graph-max-bs-decode` back up, raising `mem-fraction-static` last, only once stable. ## Key Parameters - `--speculative-algorithm {EAGLE,EAGLE3,DFLASH,STANDALONE,NGRAM,NEXTN}` — selects the method. - `--speculative-draft-model-path`, `--speculative-num-steps`, `--speculative-eagle-topk`, `--speculative-num-draft-tokens` — the three-way draft-depth/breadth/verification-capacity triad, tunable together or left to auto. - `--speculative-adaptive`, `--speculative-adaptive-config` — runtime step retuning. - `--disable-overlap-schedule` — opt out of the default V2 overlap scheduler. - `--speculative-token-map` — FR-Spec high-frequency vocabulary for EAGLE-2. ## When To Use - Situation-dependent: the docs recommend a specific method by scenario (EAGLE-3 for best speed/quality, EAGLE-2 for broad compatibility, MTP when the model has built-in multi-token heads, DFLASH/STANDALONE when a matching draft checkpoint exists, NGRAM when no extra model is available) rather than treating speculative decoding as a universal win — draft overhead, acceptance rate, batch size, hardware, and model compatibility all affect whether it helps. In the cited upstream benchmark (LLaMA-3.1-8B-Instruct, MT-Bench, 1×H100), EAGLE-3 gave a ~2.4x tok/s uplift over no speculation — an upstream reference result, not a universal expectation. - Adaptive mode specifically helps workloads whose acceptance rate shifts over time (mixed prompt difficulty, varying batch sizes); skip it if a static setting is already well tuned for a stable workload. - NGRAM is the fallback when no draft/EAGLE checkpoint exists at all. ## Risks & Pitfalls - The overlap scheduler (default for V2 speculative workers) only works with `--speculative-eagle-topk 1` — omitting `--speculative-eagle-topk` and letting auto-tuning pick `topk > 1` for some models can silently misconfigure the run without an immediate error. - DFLASH, STANDALONE, and NGRAM are each incompatible with `--enable-dp-attention`; DFLASH also requires `pp_size == 1` and disables the overlap scheduler and mixed chunked prefill, as does NGRAM. - NGRAM requires `--attention-backend flashinfer` specifically when both `page_size > 1` and BFS breadth implies `eagle_topk > 1`, or the server errors. - The three draft-depth/breadth/verification-capacity settings (`--speculative-num-steps`, `--speculative-eagle-topk`, `--speculative-num-draft-tokens`) jointly drive extra VRAM consumption (no specific multiplicative relationship is documented) and are the primary OOM lever, ahead of general batch-size settings. - Adaptive speculative decoding silently falls back to static settings outside its supported combination (EAGLE/EAGLE3 + `eagle-topk==1`) — it will not error, so verify the algorithm/topk before assuming adaptive behavior is active. - Tier switches in adaptive mode only occur between speculative rounds, never mid-round — expect one round's latency at the old tier even immediately after a switch decision. ## Related Concepts - [[concepts/parallelism-and-disaggregation]] — DCP × speculative decoding composition (draft KV cache is fully replicated per DCP rank); PD disaggregation's decode side is where speculative decoding runs. - [[concepts/hierarchical-caching]] — HiCache L2 composition with DCP explicitly excludes speculative decoding from that specific combination; HiSparse's automatic shared-index prefetch is disabled when speculative decoding is active. - [[concepts/attention-backends-and-cuda-graph]] — `--speculative-draft-attention-backend` and CUDA graph capture per speculative tier depend on backend choice. - [[concepts/quantization]] — `--speculative-draft-model-quantization` lets the draft model use a different (or no) quantization scheme than the target. ## Sources - raw/github_doc-docs-docs-advanced-features-speculative-decoding-mdx.md - raw/github_doc-docs-docs-advanced-features-adaptive-speculative-decoding-md.md - raw/github_doc-docs-docs-advanced-features-dcp-mdx.md - raw/github_doc-docs-docs-advanced-features-hisparse-guide-mdx.md --- title: "Structured Outputs and Tool Calling" type: concept tags: [structured-outputs, api, advanced, well-established, user] created: 2026-08-24 updated: 2026-08-24 sglang_version: "v0.5.18" sources: - "raw/github_doc-docs-docs-advanced-features-structured-outputs-mdx.md" - "raw/github_doc-docs-docs-advanced-features-structured-outputs-for-reasoning.md" - "raw/github_doc-docs-docs-advanced-features-tool-parser-mdx.md" - "raw/github_doc-docs-docs-advanced-features-separate-reasoning-mdx.md" confidence: medium --- # Structured Outputs and Tool Calling ## Definition SGLang constrains generation to a guaranteed output shape — JSON schema, regex, EBNF grammar, or a "structural tag" mixing free text with schema-constrained spans — via pluggable grammar backends. This grammar machinery is one of **three related but architecturally distinct subsystems** covered here: (1) **structured outputs** (the grammar-constraint engine itself), (2) **tool-call parsers** (model-family-specific decoders that turn raw model output into structured `tool_calls` objects — independent code, not grammar-driven, except that `tool_choice` specifically is implemented via EBNF grammar), and (3) **reasoning-content separation** (parser-driven splitting of `...`-style chain-of-thought from the final answer). These three interact only for the specific "exempt the thinking span from grammar constraints" feature, described below — they are not one shared mechanism. As documented in the SGLang docs, verified against `main` fetched 2026-08-24. ## How It Works ### Structured outputs: constraint types and backends Exactly one constraint parameter (`json_schema`, `regex`, or `ebnf`) may be set per request. Three grammar backends, selected via `--grammar-backend`: - **XGrammar** (default) — JSON schema, regex, EBNF (GGML BNF format); recommended for best performance/utility. - **Outlines** (`--grammar-backend outlines`) — JSON schema and regex only. - **Llguidance** (`--grammar-backend llguidance`) — JSON schema, regex, EBNF. Tip: explicitly instruct the model in the prompt to produce the desired format (e.g. "Please generate the output in the following JSON format: ...") — the grammar constrains *validity*, not quality of content. **JSON** — via Pydantic (`response_format={"type": "json_schema", "json_schema": {"name": ..., "schema": Model.model_json_schema()}}`) or a raw JSON-schema dict; native API/offline engine use the `json_schema` sampling param directly (as a JSON string). **EBNF** — via `extra_body={"ebnf": grammar_string}` (OpenAI-compatible API) or the `ebnf` sampling param (native/offline). Example: ``` root ::= city | description city ::= "London" | "Paris" | "Berlin" | "Rome" description ::= city " is " status status ::= "the capital of " country country ::= "England" | "France" | "Germany" | "Italy" ``` **Regex** — via `extra_body={"regex": "(Paris|London)"}` or the `regex` sampling param. **Structural Tag** — mixes free-form text with schema-constrained spans triggered by a marker string, useful for constraining only the tool-call portion of a response while leaving surrounding text free. Two JSON shapes are supported: the legacy `{"type": "structural_tag", "structures": [{"begin": ..., "schema": ..., "end": ...}], "triggers": [...]}` and XGrammar's newer `{"type": "structural_tag", "format": {"type": "triggered_tags", "triggers": [...], "tags": [{"begin", "content": {"type": "json_schema", "json_schema": ...}, "end"}], "at_least_one": bool, "stop_after_first": bool}}`. Available via `response_format` (OpenAI API), the `structural_tag` sampling param (native API/offline engine). All three surfaces — OpenAI-compatible chat completions, the native `/generate` SRT endpoint, and the offline `sgl.Engine` API — support all four constraint types with equivalent parameters (`response_format`/`extra_body` for OpenAI API; `sampling_params["json_schema"/"ebnf"/"regex"/"structural_tag"]` for native/offline). ### Structured outputs for reasoning models Reasoning models emit free-form `...` sections before a final answer; naively applying a grammar constraint across the whole output would force the *reasoning* text into the target schema too. SGLang instead disables grammar restrictions specifically within the reasoning span, using `--reasoning-parser` (which determines the think-end token, e.g. ``) to locate that boundary — the grammar only binds to the content *after* the reasoning ends. **This grammar-exemption feature currently supports only the DeepSeek R1 series and QwQ** (both use ``/``) — it is narrower than the general reasoning-content-separation feature below, which parses many more model families but does not by itself exempt their thinking spans from grammar constraints. ```bash python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \ --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning ``` OpenAI-compatible responses then expose `response.choices[0].message.reasoning_content` (the CoT) separately from `.content` (the schema-constrained final answer) — same `response_format`/`extra_body` constraint syntax as plain structured outputs (JSON, EBNF, regex, structural tag) all work unchanged, just scoped to the post-think portion. **Native API caveat**: set `"require_reasoning": True` in the `/generate` request body to force the model to think before producing the structured output — this is *not* needed for the chat-completions API, only the raw native endpoint. Without it, the model may skip straight to the constrained output and never reason. Offline engine: pass `reasoning_parser="deepseek-r1"` to `sgl.Engine(...)`. ### Reasoning-content separation (`--reasoning-parser`) Distinct from grammar exemption above, `--reasoning-parser` also generically splits *any* reasoning model's output into `reasoning_content` and `content` fields, independent of whether structured outputs are in use at all. | Model | Reasoning tags | Parser | Notes | |---|---|---|---| | Apertus 2509 | `<\|inner_prefix\|>` … `<\|inner_suffix\|>` | `apertus2509` | Supports `enable_thinking` | | DeepSeek-R1 series | `` … `` | `deepseek-r1` | Covers R1, R1-0528, R1-Distill (R1 itself omits the opening `` tag; R1-0528 emits both) | | DeepSeek-V3 series | `` … `` | `deepseek-v3` | Covers V3.1/V3.2 (hybrid think/non-think); use the `thinking` param, **not** `enable_thinking` | | Standard Qwen3 | `` … `` | `qwen3` | Supports `enable_thinking` | | Qwen3-Thinking | `` … `` | `qwen3` or `qwen3-thinking` | Always generates thinking content | | Kimi K2 Thinking | `◁think▷` … `◁/think▷` | `kimi_k2` | Also needs `--tool-call-parser kimi_k2` for tool use | | GPT-OSS | `<\|channel\|>analysis<\|message\|>` … `<\|end\|>` | `gpt-oss` | | Note: SGLang's own documentation is inconsistent here — some pages document `kimi`, others `kimi_k2`. The runtime's actual accepted enum (`launch_server --help` / `server_args.py`) is authoritative; verify against your installed version. OpenAI-compatible API contract follows the DeepSeek reasoning-model API design: `reasoning_content` = CoT, `content` = final answer. Reasoning separation is **on by default** once `--reasoning-parser` is set; disable per-request with `extra_body={"separate_reasoning": False}`. For streaming, `stream_reasoning: False` buffers all reasoning content to the last reasoning chunk (or first post-reasoning chunk) instead of streaming it token-by-token. Native API: POST the raw generated text plus `{"reasoning_parser": "deepseek-r1"}` to `/separate_reasoning` to split it after the fact. Offline engine: `sglang.srt.parser.reasoning_parser.ReasoningParser("deepseek-r1").parse_non_stream(generated_text)`. To support a new reasoning model schema, subclass `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py`. ### Tool / function calling Enable with `--tool-call-parser ` at server launch; the parser determines how raw model output is decoded into structured `tool_calls`. Supported parsers: | Parser | Models | Notes | |---|---|---| | `apertus2509` | Apertus 2509 (`swiss-ai/Apertus-{8,70}B-Instruct-2509`) | Tool calls as JSON list of single-key objects: `<\|tools_prefix\|>[{"tool": {...}}]<\|tools_suffix\|>` | | `deepseekv3` | DeepSeek-V3 (`deepseek-ai/DeepSeek-V3-0324`) | Recommend `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` | | `deepseekv31` | DeepSeek-V3.1, V3.2-Exp | Recommend matching `tool_chat_template_deepseekv31.jinja`/`...v32.jinja` | | `deepseekv32` | DeepSeek-V3.2 | | | `glm` | GLM series (e.g. `zai-org/GLM-4.6`) | | | `gpt-oss` | GPT-OSS (`openai/gpt-oss-120b`/`-20b`, bf16 variants) | Filters out analysis-channel events, keeps only normal text — content can end up empty if explanations live in the analysis channel; work around by completing the tool round with a `role="tool"` message so the model produces final content | | `kimi_k2` | `moonshotai/Kimi-K2-Instruct` | | | `llama3` | Llama 3.1/3.2/3.3 | | | `llama4` | Llama 4 (e.g. `Llama-4-Scout-17B-16E-Instruct`) | | | `mistral` | Mistral 7B/Nemo variants | | | `pythonic` | Llama-3.2/3.3/4 | Model emits function calls as literal Python code (see below) | | `qwen` | Qwen series except Qwen3-Coder (e.g. `Qwen3-Next-80B-A3B-Instruct`, `Qwen3-VL-30B-A3B-Thinking`) | | | `qwen3_coder` | Qwen3-Coder | | | `step3` | Step-3 | | Note: this table (reproduced from SGLang's own tool-parser doc) omits `qwen25`, even though the example command below uses `--tool-call-parser qwen25` — an upstream table-incompleteness gap, not a wiki error. Also note the same `kimi`/`kimi_k2` naming inconsistency described above applies to the tool-call parser name. ```bash python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning ``` OpenAI-compatible flow: pass `tools=[...]` to `chat.completions.create(...)`; the response's `message.tool_calls[i].function.{name,arguments}` carries the parsed call (streaming: accumulate `delta.tool_calls` chunks and concatenate `.function.arguments` fragments into one JSON string). After executing the tool, append a `{"role": "tool", "tool_call_id": ..., "content": str(result), "name": tool_name}` message and re-call the model to get the final answer. Native API: call `/generate` normally with `skip_special_tokens: False` (so tool-call delimiter tokens survive), then POST the raw text plus `{"tool_call_parser": "qwen25", "tools": [...]}` to `/parse_function_call`, which returns `{"normal_text": ..., "calls": [{"name": ..., "parameters": ...}, ...]}`. Offline engine: use `sglang.srt.function_call.function_call_parser.FunctionCallParser(tools=..., tool_call_parser="qwen25").parse_non_stream(generated_text)`. Note: for the `gpt-oss` tool parser specifically, add `"no_stop_trim": True` to sampling params so the tool-call token isn't trimmed. **Tool choice mode**: SGLang supports OpenAI's `tool_choice`, implemented via EBNF grammar for reliability. `tool_choice="required"` forces at least one tool call; `tool_choice={"type": "function", "function": {"name": "specific_fn"}}` forces a specific function. Fully supported with the **XGrammar** backend (default); other backends like `outlines` may not fully support it. **Pythonic tool call format** (Llama-3.2/3.3/4): model emits calls as literal Python syntax, e.g. `[get_current_weather(city="San Francisco", state="CA", unit="celsius")]` — a Python list, arguments as Python literals (not JSON), supporting multiple parallel calls in one list. Enable with `--tool-call-parser pythonic`, and strongly recommend also setting a matching `--chat-template` (e.g. `examples/chat_template/tool_chat_template_llama4_pythonic.jinja`) since the model expects a specific prompt structure (special tokens, `<|eom|>` boundaries) to reliably emit valid pythonic output — without it, tool calling may fail or be inconsistent, and heavy prompt engineering becomes the only lever. Still under active development on Blackwell. A model heavily finetuned on JSON tool calls may default back to JSON regardless of instructions. **Adding a new model's tool parser**: (1) add the model's tool-call tags (e.g. ``, `[TOOL_CALLS]`) to `TOOLS_TAG_LIST` in `sglang/srt/function_call_parser.py`; (2) create a new `BaseFormatDetector` subclass handling that format; (3) register it in `MultiFormatParser`. ## Key Parameters - `--grammar-backend {xgrammar,outlines,llguidance}` — structured-output constraint engine. - `json_schema` / `regex` / `ebnf` — three alternative constraint payload shapes; exactly one may be set per request. `structural_tag` is a fourth, separate constraint shape (its own sampling param / `response_format` value) — the docs do not describe combining it with the other three on the same request. - `--reasoning-parser {deepseek-r1,deepseek-v3,qwen3,qwen3-thinking,kimi_k2,gpt-oss,apertus2509}` — reasoning-content splitting for any listed model family; grammar-exemption for the thinking span specifically is currently supported only for DeepSeek R1 series and QwQ. - `separate_reasoning` (request-level, default effectively True once `--reasoning-parser` is set), `stream_reasoning` — control reasoning-content extraction per request. - `require_reasoning` (native `/generate` only) — forces thinking before constrained output. - `--tool-call-parser ` — function-calling output parser. - `tool_choice`, `tools` (OpenAI API request fields) — tool-call control. - `--chat-template` — recommended specifically alongside the `deepseekv3`/`deepseekv31`/`deepseekv32` tool-call parsers and the `pythonic` tool-call format to match those models' expected prompt structure; not a general requirement for every tool or reasoning parser. ## When To Use - Structured outputs: any workload needing guaranteed-parseable output (JSON APIs, enum-constrained answers, DSL-constrained generation). - Structural tags specifically: mixed free-text + tool-call outputs, where only the tool-call span needs schema enforcement. - `--reasoning-parser`: any reasoning model (R1, QwQ, Qwen3, Kimi K2, GPT-OSS, Apertus) where the caller wants CoT separated from the final answer; the additional "structured outputs apply only after reasoning completes" grammar-exemption behavior is currently limited to DeepSeek R1 series and QwQ. - `--tool-call-parser`: any agentic/function-calling deployment; must match the specific model family in use. ## Risks & Pitfalls - Only one of `json_schema`/`regex`/`ebnf` may be set per request — combining them is not supported. `structural_tag` is a separate constraint shape, not documented as combinable with the other three either. - Outlines backend doesn't support EBNF and may not fully support `tool_choice` — XGrammar is the safer default for both. - Native `/generate` structured outputs on reasoning models may skip straight to the constrained output and never reason unless `require_reasoning: True` is explicitly set — this pitfall does not exist on the chat-completions API. - DeepSeek-V3.1/V3.2 use the `thinking` parameter, not `enable_thinking` like Qwen3/Apertus — an easy naming mix-up between model families. - `gpt-oss` tool parser strips analysis-channel content, which can leave `content` empty when the model puts its explanation there — requires the tool round-trip (`role="tool"` message) to recover final text, and requires `no_stop_trim: True` in offline-engine sampling params to avoid clipping the tool-call token. - Pythonic tool calling without the matching chat template is fragile — the model may silently fall back to JSON if heavily finetuned on that format, regardless of prompt instructions. - Kimi K2 and Apertus 2509 require setting **both** `--reasoning-parser` and a matching `--tool-call-parser` for correct combined reasoning + tool-use behavior — setting only one is a common gap. ## Related Concepts - [[concepts/quantization]] and [[concepts/attention-backends-and-cuda-graph]] are largely orthogonal to this page but interact at the server-launch-flag level (all are `sglang.launch_server` arguments). - [[concepts/lora-and-model-loading]] — chat templates referenced here (`--chat-template`) are also relevant when loading models with non-default prompt formats. ## Sources - raw/github_doc-docs-docs-advanced-features-structured-outputs-mdx.md - raw/github_doc-docs-docs-advanced-features-structured-outputs-for-reasoning.md - raw/github_doc-docs-docs-advanced-features-tool-parser-mdx.md - raw/github_doc-docs-docs-advanced-features-separate-reasoning-mdx.md --- title: "Supported Hardware" type: concept tags: [hardware, install, operator, foundational] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-hardware-platforms-overview-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-nvidia-gpus-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-amd-gpu-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-apple-metal-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-cpu-server-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-mthreads-gpu-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-nvidia-jetson-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-tpu-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-xpu-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-plugin-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-getting-started-ins.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-getting-started-qui.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-faq-mdx.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-glossary-.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-environme.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-support-m.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-support-f.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-parame.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-quanti.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-profil.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-ring-s.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-evaluation-accuracy.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-evaluation-performa.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-diffusion-disaggreg.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-mindspore-backend-m.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-development-contrib.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-development-operato.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-development-support.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-model-deployment-be.md", "raw/github_doc-docs-docs-hardware-platforms-ascend-npus-model-deployment-tu.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition SGLang runs across a wide hardware landscape: NVIDIA GPUs (in this KB's assessment, the primary, most feature-complete target), AMD GPUs (ROCm/HIP), Huawei Ascend NPUs (a large, independently-maintained documentation and code tree), Google TPUs (via a separate JAX-based sibling project, SGLang-JAX), Intel GPUs (XPU) and CPUs (Xeon AMX), Apple Silicon (native MLX backend), NVIDIA Jetson (edge), Moore Threads GPUs (MUSA), and an out-of-tree hardware plugin system for vendors who don't want to touch the main repo at all. The raw hardware-platforms doc set is dominated by Ascend NPU material (50 of the 60 hardware-platform files in this KB's raw corpus — a corpus-inventory observation, not a figure the overview page itself states), reflecting how much dedicated tooling that platform needs relative to the others, which are mostly single-page guides (raw/github_doc-docs-docs-hardware-platforms-overview-mdx.md). *Note: as documented in the SGLang docs, verified against `main` fetched 2026-08-24 — these are mutable upstream docs, not tag-pinned to v0.5.18.* ## How It Works Every mainline SGLang platform converges on the same launch surface — `python3 -m sglang.launch_server` / `sglang serve` with a `--model-path` — but the `--device` and `--attention-backend` flags, the install method, and the available quantization/parallelism features diverge sharply by vendor. Google TPU is the exception: it runs through the separate SGLang-JAX project with its own `sgl_jax.launch_server` entry point, not `sglang.launch_server` (see below). - **NVIDIA GPUs** are the default target. The platform page for NVIDIA GPUs simply redirects to the general [[concepts/installation]] guide (raw/github_doc-docs-docs-hardware-platforms-nvidia-gpus-mdx.md) — no special flags are needed, and this is the baseline every other platform is compared against. - **AMD GPUs (ROCm)** need system tuning before install: append `pci=realloc=off iommu=pt` to `GRUB_CMDLINE_LINUX` and disable NUMA auto-balancing (`echo 0 > /proc/sys/kernel/numa_balancing`). Install from source (`git clone -b v0.5.18`, build `sgl-kernel` via `setup_rocm.py`, `pip install -e "python[all_hip]"`) or via the `rocm.Dockerfile` image on Docker Hub (`lmsysorg/sglang`). Kernels accelerated by [Aiter](https://github.com/ROCm/aiter) are enabled with `export SGLANG_USE_AITER=1` (raw/github_doc-docs-docs-hardware-platforms-amd-gpu-mdx.md). - **Apple Silicon (Metal/MLX)** requires macOS 14+, PyTorch 2.13.x, and MLX ≥0.32.0 (installed via the `srt_mps` extra), plus a full Xcode install (not just Command Line Tools) if you want to build the optional native Metal kernels in `sgl-kernel`. The runtime is enabled with `SGLANG_USE_MLX=1`; without it SGLang falls back to `torch.mps`. `--disable-cuda-graph` is required since CUDA graphs don't apply (raw/github_doc-docs-docs-hardware-platforms-apple-metal-mdx.md). - **CPU servers (Intel Xeon AMX)** need 4th-gen-or-newer Xeon Scalable processors. Install via the `xeon`-suffixed Docker image (`lmsysorg/sglang:v0.5.13-xeon`) or from source using `uv` and a dedicated `pyproject_cpu.toml`. Requires `export SGLANG_USE_CPU_ENGINE=1` and correct `LD_LIBRARY_PATH`/`LD_PRELOAD` for `libiomp5`/`libtcmalloc`/`libtbbmalloc`. On CPU, one tensor-parallel rank corresponds to one sub-NUMA cluster (SNC); `SGLANG_CPU_OMP_THREADS_BIND` explicitly pins CPU core ranges to each TP rank (raw/github_doc-docs-docs-hardware-platforms-cpu-server-mdx.md). - **Moore Threads GPUs (MUSA)** support only a from-source install: build `sgl-kernel` via `setup_musa.py`, then `pip install -e "python[all_musa]"` (raw/github_doc-docs-docs-hardware-platforms-mthreads-gpu-mdx.md). - **NVIDIA Jetson Orin** runs via the community `jetson-containers` project (JetPack 6.1+, high-performance mode via `nvpmodel -m 0`). Limited on-device compute forces `--dtype half` and a reduced `--context-length` (e.g. 8192) (raw/github_doc-docs-docs-hardware-platforms-nvidia-jetson-mdx.md). - **Google TPU** is not served by mainline SGLang at all — it goes through **SGLang-JAX**, a separate JAX-native repo/package (`pip install sglang-jax`) with its own launcher (`sgl_jax.launch_server`), its own bench tools (`sgl_jax.bench_serving`), and Python 3.12+. It supports TPU v6e/v7, FlashAttention and native attention backends (FlashAttention is recommended for production), continuous batching, radix-tree prefix caching, TP, chunked prefill, and EAGLE/EAGLE3 speculative decoding; quantization, data-parallel attention, and multi-LoRA are still in development (🚧) (raw/github_doc-docs-docs-hardware-platforms-tpu-mdx.md). - **Intel GPUs (XPU)**: the docs describe source install as the currently supported dependency path — create a conda env, install PyTorch/torchvision/torchaudio from the `xpu` index, then build with `pyproject_xpu.toml` — but also document a Dockerfile-based container workflow (`docker build -t sglang-xpu:latest -f xpu.Dockerfile .`). Launch with `--device xpu` and the XPU-specific `--attention-backend intel_xpu` (page size constrained to `[32, 64, 128]`). XPU has its own opt-in graph-capture system analogous to CUDA graph (`--cuda-graph-backend-decode full`, `--cuda-graph-backend-prefill {tc_piecewise,breakable}`), and supports PD disaggregation via the NIXL transfer backend. `--enable-memory-saver` and `--enable-two-batch-overlap` are **not yet supported** on XPU (raw/github_doc-docs-docs-hardware-platforms-xpu-mdx.md). - **Ascend NPU** is, in this KB's assessment, the deepest platform integration outside NVIDIA, with its own installation/quickstart/FAQ/glossary/reference/optimization/evaluation/development doc subtree plus per-model tutorial and best-practice pages. See "Ascend NPU in depth" below. - **Out-of-tree hardware plugin system**: any vendor can add a new device without touching sglang's source by registering a `SRTPlatform` subclass under the `sglang.srt.platforms` entry-point group (selected via `SGLANG_PLATFORM`) and/or a general behavior-injection plugin under `sglang.srt.plugins` (whitelisted via `SGLANG_PLUGINS`). A `DeviceMixin` base class supplies identity queries (`is_cuda()`, `is_rocm()`, `is_npu()`, `is_xpu()`, `is_musa()`, `is_cuda_alike()`) and device operations; `current_platform` is a lazy singleton that auto-discovers the active platform at first use (raw/github_doc-docs-docs-hardware-platforms-plugin-mdx.md). ### Ascend NPU in depth Ascend support centers on two hardware SKUs, distinguished throughout the docs as **A2** (Atlas 800I A2, Ascend 910B, 8 devices, 1 die/card, 64GB/die) and **A3** (Atlas 800I A3, Ascend 910C, 16 devices, 2 dies/card) — A3's extra die-per-card doubles the effective `--tp-size` for the same card count (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-glossary-.md). The software stack is a pinned component matrix (HDK, CANN, TorchNPU, MemFabric/MemFabric-zbal for PD KV transfer, Triton-Ascend, and the `sgl-kernel-npu` operator package — e.g. HDK 25.5.2 / CANN 9.0.0 / TorchNPU 26.0.0 for this doc snapshot) obtained either by installing from source against a pulled CANN Docker image, or by pulling a prebuilt `quay.io/ascend/sglang:cann-{a3|910b}-v` image — stable-release and daily-build tags both exist (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-getting-started-ins.md, raw/github_doc-docs-docs-hardware-platforms-ascend-npus-getting-started-qui.md, raw/github_doc-docs-docs-hardware-platforms-ascend-npus-faq-mdx.md). Ascend has its own attention backend (`--attention-backend ascend`, `--mm-attention-backend ascend_attn` for multimodal), its own PD-disaggregation transfer backend (`--disaggregation-transfer-backend ascend`, using the MemFabric-Hybrid drop-in replacement for Mooncake), NPU Graph (the CUDA-graph analog, `torch.npu.NPUGraph`), and a dedicated quantization stack (`--quantization modelslim`, supporting W4A4/W4A8/W8A8/W4A16, auto-detected from `quant_model_description.json`; newer Ascend 950-series hardware additionally supports MXFP8/MXFP4 online and offline paths not available on A2/A3) (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-quanti.md). Note: the AMD/Ascend/MUSA vendor docs quoted above still say `sgl-kernel` in their build commands; the actual package was renamed `sgl-kernel` → `sglang-kernel` in v0.5.10 (raw/github_release-v0-5-10.md), so the docs retain the legacy name while this KB targets v0.5.18. A large set of Ascend-only environment variables tune MLA fusion (`SGLANG_NPU_USE_MLAPO`, `SGLANG_USE_FIA_NZ`), MoE dual-stream execution (`SGLANG_NPU_USE_MULTI_STREAM`), and DeepEP dispatch (`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`, `DEEPEP_HCCL_BUFFSIZE`) (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-environme.md). A companion parameter-tuning guide documents required-vs-optimization-only flags for TP/DP/EP/CP/PD, with concrete reference values pulled from the DeepSeek-V3.2 best-practice recipe (e.g. `--mem-fraction-static` 0.73 prefill / 0.79 decode, `HCCL_BUFFSIZE` 1200 prefill / 400 decode) (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-parame.md). Beyond the reference docs, Ascend ships **per-model deployment tutorials** (environment setup, feature walkthroughs, e.g. `docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx`) and **best-practice pages** (tuned launch commands with measured TPOT for named deployment shapes like "DeepSeek-R1 W8A8 2P1D 32P", e.g. `.../best-practices/deepseek_r1.mdx`) for models including DeepSeek-R1/V3.2, Qwen3/3.5, and others — this is where most of the 50 Ascend files live; only the representative DeepSeek-R1 and DeepSeek-V3.2 pages are individually cited in this page's sources, the rest of the catalog was sampled by filename/grep rather than individually read in full (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-model-deployment-be.md, raw/github_doc-docs-docs-hardware-platforms-ascend-npus-model-deployment-tu.md). Ascend also has: an accuracy/performance evaluation guide built on EvalScope and AISBench (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-evaluation-accuracy.md, .../evaluation-performa.md); a profiling guide reusing SGLang's PyTorch-Profiler workflow with Ascend-specific caveats (trace merging across nodes is unreliable on the `*_ascend_pt` format) (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-profil.md); a PD-diffusion disaggregation path via a from-source Mooncake build with `-DUSE_ASCEND_DIRECT=ON` (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-diffusion-disaggreg.md); an alternate **MindSpore execution backend** (`sgl-mindspore` package, `--model-impl mindspore`, currently Qwen3 and DeepSeek V3/R1 only) (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-mindspore-backend-m.md); and its own operator-development guide for contributing Ascend C / Triton kernels to the separate `sgl-kernel-npu` repo, plus a contribution guide covering NPU-specific CI registration and model onboarding (raw/github_doc-docs-docs-hardware-platforms-ascend-npus-development-operato.md, .../development-contrib.md, .../development-support.md). ## Key Parameters - **`--device`** — `cuda` (default), `rocm`/HIP is auto-detected, `npu`, `xpu`, `cpu`, `mps`; on Ascend, `--device` and `--attention-backend` are auto-detected once `torch.npu.is_available()` is true, so they can be omitted. - **`--attention-backend`** — platform-specific values: `ascend` (NPU), `intel_xpu` (XPU), `flashinfer`/`fa`/`triton` families (NVIDIA/AMD); on SGLang-JAX (TPU) the choices are `fa` or `native`. - **Vendor env-var families** — `SGLANG_USE_AITER` (AMD), `SGLANG_USE_MLX` / `SGLANG_MLX_*` (Apple), `SGLANG_USE_CPU_ENGINE` / `SGLANG_CPU_OMP_THREADS_BIND` (Xeon CPU), `SGLANG_NPU_*` / `ASCEND_*` (Ascend), `JAX_COMPILATION_CACHE_DIR` (TPU/SGLang-JAX). These are not interchangeable across platforms. - **Quantization availability is vendor-specific**, not a single matrix: FP8/AWQ/MXFP4/GPTQ/compressed-tensors/Quark/`petit_nvfp4` all work on AMD, but Marlin- and NVIDIA-specific kernels (`awq_marlin`, `gptq_marlin`, `gguf`, `modelopt_fp8`, `modelopt_fp4`) do not; Ascend has its own `modelslim` scheme plus hardware-gated MXFP8/MXFP4 support that requires Ascend 950-series silicon; SGLang-JAX has no quantization yet. - **Docker image tags encode platform + version**: e.g. `lmsysorg/sglang:v0.5.13-xeon` (CPU), `sglang_image` built from `rocm.Dockerfile`/`npu.Dockerfile`/`xpu.Dockerfile` (AMD/Ascend/XPU), `quay.io/ascend/sglang:cann9.0.0-{a3|910b}-v0.5.16` (Ascend, CANN-version-qualified). ## When To Use Use NVIDIA GPUs as the default unless a specific constraint pushes elsewhere: AMD ROCm for existing AMD fleets (MI300X/MI355X etc.) with the caveat that some quantization kernels are unavailable; Ascend NPU for Huawei Cloud/on-prem Atlas 800I deployments — in this KB's assessment the richest non-NVIDIA path, given its dedicated tuning docs and per-model best practices; SGLang-JAX for Google Cloud TPU workloads specifically; XPU/CPU for Intel-hardware-constrained environments; Apple Silicon or Jetson for local/edge development and testing rather than production-scale serving; and the plugin system for any hardware vendor not already in tree. ## Risks & Pitfalls - Treating quantization or feature flags as portable across vendors is a common mistake — e.g. `--quantization gguf` or Marlin-backed kernels (`awq_marlin`, `gptq_marlin`, `modelopt_fp8`, `modelopt_fp4`) do not work on AMD; MXFP8/MXFP4 require specific Ascend hardware generations (950-series), not A2/A3. - SGLang-JAX (TPU) is, based on documented feature coverage, markedly behind the CUDA/ROCm/NPU paths — quantization and multi-LoRA are explicitly marked "in development." - XPU lacks `--enable-memory-saver` and `--enable-two-batch-overlap`, and speculative decoding is "not yet implemented" there. - Ascend's software stack is version-pinned tightly (HDK/CANN/TorchNPU/Triton-Ascend/sgl-kernel-npu must line up); the Ascend FAQ documents this as a specific example risk rather than a quantified failure rate — e.g. one graph-mode kernel error resolved only by matching CANN+TorchNPU versions, or by capping captured-graph count to 10 or fewer. - Apple MLX quantization is fixed at group size 64 and only supports 4-bit/8-bit; it silently no-ops if the model is already quantized in its HF config. - Jetson's limited memory forces `--dtype half` and a small `--context-length`, which is easy to overlook when copying commands from the general quickstart. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/installation]] - [[concepts/quantization]] - [[concepts/parallelism-and-disaggregation]] - [[concepts/server-arguments]] - [[concepts/attention-backends-and-cuda-graph]] (planned) ## Sources - raw/github_doc-docs-docs-hardware-platforms-overview-mdx.md - raw/github_doc-docs-docs-hardware-platforms-nvidia-gpus-mdx.md - raw/github_doc-docs-docs-hardware-platforms-amd-gpu-mdx.md - raw/github_doc-docs-docs-hardware-platforms-apple-metal-mdx.md - raw/github_doc-docs-docs-hardware-platforms-cpu-server-mdx.md - raw/github_doc-docs-docs-hardware-platforms-mthreads-gpu-mdx.md - raw/github_doc-docs-docs-hardware-platforms-nvidia-jetson-mdx.md - raw/github_doc-docs-docs-hardware-platforms-tpu-mdx.md - raw/github_doc-docs-docs-hardware-platforms-xpu-mdx.md - raw/github_doc-docs-docs-hardware-platforms-plugin-mdx.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-getting-started-ins.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-getting-started-qui.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-faq-mdx.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-glossary-.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-environme.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-support-m.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-reference-support-f.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-parame.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-quanti.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-profil.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-optimization-ring-s.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-evaluation-accuracy.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-evaluation-performa.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-diffusion-disaggreg.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-mindspore-backend-m.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-development-contrib.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-development-operato.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-development-support.md - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-model-deployment-be.md (representative best-practice page; ~13 similar per-model chunks not individually cited) - raw/github_doc-docs-docs-hardware-platforms-ascend-npus-model-deployment-tu.md (representative tutorial page; ~16 similar per-model chunks not individually cited) --- title: "Supported Models" type: concept tags: [models, api, foundational, user] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_doc-docs-docs-supported-models-mdx.md", "raw/github_doc-docs-docs-supported-models-generative-models-mdx.md", "raw/github_doc-docs-docs-supported-models-multimodal-language-models-mdx.md", "raw/github_doc-docs-docs-supported-models-embedding-models-mdx.md", "raw/github_doc-docs-docs-supported-models-reward-models-mdx.md", "raw/github_doc-docs-docs-supported-models-rerank-models-mdx.md", "raw/github_doc-docs-docs-supported-models-classify-models-mdx.md", "raw/github_doc-docs-docs-supported-models-diffusion-language-models-mdx.md", "raw/github_doc-docs-docs-supported-models-mindspore-models-mdx.md", "raw/github_doc-docs-docs-supported-models-modelscope-mdx.md", "raw/github_doc-docs-docs-supported-models-support-new-models-mdx.md", "raw/github_doc-docs-docs-supported-models-transformers-fallback-mdx.md", "raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md"] confidence: medium sglang_version: "v0.5.18" --- ## Definition SGLang serves model families across six categories: large language models (text-in/text-out, including MoE architectures), multimodal/vision-language models, diffusion (non-autoregressive) language models, embedding models, rerank models, and reward/classification models (raw/github_doc-docs-docs-supported-models-mdx.md). Coverage is broad by design — dozens of LLM families are supported natively, community models load via a generic `transformers` fallback, and models not yet built into SGLang can be added without touching SGLang's source at all. ## How It Works Each category has its own launch pattern built on the same `sglang.launch_server` / `sglang serve` entry point: - **Large language models** (raw/github_doc-docs-docs-supported-models-generative-models-mdx.md): the default case — `python3 -m sglang.launch_server --model-path `. Coverage spans frontier families (DeepSeek v1/v2/v3/R1, Kimi K2/K2 Thinking/Linear, GPT-OSS, Qwen 3.5/3/3MoE/3Next/2.5/2, Llama 2/3.x/4, Mistral/Mixtral, Gemma 1/2/3, Phi 1.5–4/MoE) through many smaller or specialized entries (MiniCPM, OLMo/OLMoE, MiniMax-M2 family, StableLM, Command-R/A, DBRX, Grok, ChatGLM, InternLM2, ExaONE 3, Baichuan 2, XVERSE, SmolLM, GLM-4, MiMo, ERNIE-4.5, Arcee AFM/Trinity, Nemotron Super/Ultra/Nano/3, StarCoder2, Jet-Nemotron, LFM2/LFM2-MoE, Falcon-H1, Hunyuan-Large, IBM Granite 3.0/3.1/4.0/SWA, Sarvam 2, Laguna XS.2, Mellum 2, and more). If unsure whether an architecture is implemented, GitHub-search the class name (e.g. `Qwen3ForCausalLM`) scoped to `python/sglang/srt/models/`. - **Multimodal (vision/audio/video) models** (raw/github_doc-docs-docs-supported-models-multimodal-language-models-mdx.md): launched the same way, with `--enable-multimodal` where required. Covers Qwen-VL family, DeepSeek-VL2/OCR/Janus-Pro, MiniCPM-V/o, Llama 3.2 Vision, LLaVA family, Gemma 3 (multimodal), Kimi-VL, Mistral-Small-3.1, Phi-4-multimodal, MiMo-VL, GLM-4.5V/4.1V, PaddleOCR-VL, GLM-OCR, DotsVLM(-OCR), NVILA, NVIDIA Nemotron Nano 2.0 VL, Ernie4.5-VL, Qwen3-ASR/Omni, LFM2-VL, and NVIDIA LocateAnything-3B (visual grounding). Audio-only ASR (Whisper, Qwen3-ASR) is served through the OpenAI-compatible `/v1/audio/transcriptions` endpoint. Video input is supported for several VLM families via `sgl.video(path, num_frames)` or an OpenAI-style `video_url` content part; frame sampling and feature merging are handled per-model (e.g. Qwen-VL's own sampler, NVIDIA Nemotron Nano's EVS token-pruning at `video_pruning_rate=0.7` by default). - **Diffusion language models** (raw/github_doc-docs-docs-supported-models-diffusion-language-models-mdx.md): non-autoregressive text generation via `--dllm-algorithm {LowConfidence,JointThreshold}` (+ optional `--dllm-algorithm-config`). First-Done-First-Out (FDFO) scheduling is on by default (each request leaves the batch as soon as its block resolves, instead of lockstep advancement); disable with `--no-dllm-fdfo`. Covers LLaDA2.0 (mini/flash), SDAR (JetLM, dense and MoE), and DiffusionGemma (renoising block-diffusion multimodal MoE). - **Embedding models** (raw/github_doc-docs-docs-supported-models-embedding-models-mdx.md): native encoder architectures and `google/embeddinggemma-300m` are auto-detected; decoder-style embedding models need `--is-embedding` (+ `--trust-remote-code` if required). Covers EmbeddingGemma (bidirectional, auto-detected, served with breakable CUDA graph by default on CUDA), E5 (Llama/Mistral-based), GTE-Qwen2, Qwen3-Embedding, bare `Qwen3Model` backbones (auto-classified as embedding), BGE (needs `triton`/`torch_native` attention backend), multimodal GME, and CLIP. Matryoshka (MRL) truncatable embeddings are supported via `matryoshka_dimensions`/`is_matryoshka` in `--json-model-override-args`, with a `dimensions` field on the request. - **Rerank models** (raw/github_doc-docs-docs-supported-models-rerank-models-mdx.md): two distinct serving modes — cross-encoder rerankers (e.g. BGE-Reranker) launch **with** `--is-embedding`; decoder-only yes/no rerankers (Qwen3-Reranker, and its multimodal sibling Qwen3-VL-Reranker for text/image/video) launch **without** `--is-embedding` and require an explicit `--chat-template` (the reranker jinja template), using next-token logprob scoring instead. Both are served via the vLLM-compatible `/v1/rerank` endpoint with `query`/`documents`/`top_n`/`return_documents` parameters. - **Reward and classification models** (raw/github_doc-docs-docs-supported-models-reward-models-mdx.md, raw/github_doc-docs-docs-supported-models-classify-models-mdx.md): reward models run with `--is-embedding` and score via the `/v1/classify`-adjacent path (Llama-3.1-Reward, Gemma-2-Reward, InternLM2-Reward, Qwen2.5-Math-RM, Qwen2.5-SequenceClassification). The dedicated `/v1/classify` endpoint (vLLM-compatible) works with classification models supported by SGLang — documented as `LlamaForSequenceClassification`, `Qwen2ForSequenceClassification`, `Qwen3ForSequenceClassification`, `BertForSequenceClassification`, and `Gemma2ForSequenceClassification` — for multi-class classification, auto-mapping labels from the model's `id2label` config, and doubles as the interface reward models use for single-score output. - **Running an unlisted/new model without modifying SGLang** (raw/github_doc-docs-docs-supported-models-support-new-models-mdx.md): (1) fall back to the generic **transformers backend** — `--model-impl transformers` reuses any HF model that implements `_supports_attention_backend = True` and forwards through `ALL_ATTENTION_FUNCTIONS`, supporting most quantization schemes except GGUF (raw/github_doc-docs-docs-supported-models-transformers-fallback-mdx.md); or (2) **register a custom model class** at runtime via `ModelRegistry.models.update(...)` for the offline `Engine` API, or via the `SGLANG_EXTERNAL_MODEL_PACKAGE` env var (plus `SGLANG_EXTERNAL_MM_MODEL_ARCH` / `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE` for multimodal) to use the standard `sglang.launch_server` CLI unmodified — the model file needs an `EntryClass` variable matching the checkpoint's `config.json` `"architectures"` field. - **ModelScope models**: set `export SGLANG_USE_MODELSCOPE=true` before launch to resolve model IDs from ModelScope instead of Hugging Face (raw/github_doc-docs-docs-supported-models-modelscope-mdx.md); ModelScope uses a separate cache directory from Hugging Face. - **MindSpore-backed models**: a fully separate execution path (`sgl-mindspore` package, `--model-impl mindspore`) rather than a model family, currently covering Qwen3 (dense and MoE) and DeepSeek V3/R1 as documented in the 2026-08-24 source snapshot ("more models coming soon") — see [[concepts/supported-hardware]] for the Ascend-specific installation details (raw/github_doc-docs-docs-supported-models-mindspore-models-mdx.md). ## Key Parameters - **`--is-embedding`** — routes non-generative model types (embeddings, cross-encoder rerankers, reward models) through the embedding runner instead of causal-LM decoding. - **`--trust-remote-code`** — required for many decoder-style embedding/rerank/OCR models that ship custom modeling code. - **`--chat-template`** — mandatory for decoder-only rerankers (Qwen3-Reranker family); omitting it makes the server treat the model as a plain LLM and reject `/v1/rerank` with a 400. - **`--enable-multimodal`** — required for specific documented paths (e.g. registering an external/custom multimodal model), but the primary launch example for a built-in VLM (Llama 3.2 Vision) omits it — check the model's own launch example rather than assuming it's universally required. - **`--model-impl {auto,sglang,transformers,mindspore}`** — selects which backend implements the forward pass for a given architecture. The server-arguments reference documents `auto`/`sglang`/`transformers` (raw/github_doc-docs-docs-advanced-features-server-arguments-mdx.md); `mindspore` is an additional value documented separately in the MindSpore-models source, not part of that same table. - **`SGLANG_EXTERNAL_MODEL_PACKAGE` / `SGLANG_EXTERNAL_MM_MODEL_ARCH` / `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`** — register out-of-tree model code with the standard CLI. - **`--keep-mm-feature-on-device`** and **`--mm-process-config`** — multimodal latency/memory tuning knobs (keep feature tensors on GPU vs. move to CPU; cap per-modality `max_pixels`/`fps`/`max_frames`). ## When To Use Reach for a native SGLang model implementation whenever one exists — they get RadixAttention, and native implementation is generally preferred, but parallelism/quantization/speculative-decoding feature compatibility and tuning maturity should be checked per model/platform rather than assumed complete. Use the `transformers` fallback for a model on the Hub that SGLang hasn't implemented natively but that follows a standard HF attention interface — this is the lowest-effort path and still gets most quantization support (multimodal support through this path is documented as "coming soon" in the 2026-08-24 snapshot). Reach for `ModelRegistry` / `SGLANG_EXTERNAL_MODEL_PACKAGE` registration when you need custom forward-pass logic (e.g. logit post-processing, a bespoke architecture) without forking SGLang or waiting for a native model PR — see [[concepts/developer-and-benchmarking]] for the accuracy/benchmark bar new models are expected to clear before being added to SGLang itself. ## Risks & Pitfalls - Launching a decoder-only reranker with `--is-embedding` breaks it — logprob-based yes/no scoring requires the causal-LM path, not the embedding runner; the docs explicitly instruct relaunching without `--is-embedding` to fix it. - Forgetting `--chat-template` on Qwen3-(VL-)Reranker produces a generic "not an embedding model" 400 error that looks unrelated to the actual cause. - BGE-family embedding/rerank models require the `triton` or `torch_native` attention backend specifically — other backends aren't supported for these architectures. - The `transformers` fallback does not support GGUF quantization, and multimodal support through that path is documented as "coming soon" as of the 2026-08-24 snapshot. - `--quantization mlx_q4`/`mlx_q8`, GGUF, and vendor-specific quant schemes (see [[concepts/supported-hardware]]) are not universally available across every model family — check the model's own doc entry before assuming a quantization flag will work. ## Related Concepts - [[concepts/sglang-overview]] - [[concepts/supported-hardware]] - [[concepts/quantization]] - [[concepts/server-arguments]] - [[concepts/developer-and-benchmarking]] ## Sources - raw/github_doc-docs-docs-supported-models-mdx.md - raw/github_doc-docs-docs-supported-models-generative-models-mdx.md - raw/github_doc-docs-docs-supported-models-multimodal-language-models-mdx.md - raw/github_doc-docs-docs-supported-models-embedding-models-mdx.md - raw/github_doc-docs-docs-supported-models-reward-models-mdx.md - raw/github_doc-docs-docs-supported-models-rerank-models-mdx.md - raw/github_doc-docs-docs-supported-models-classify-models-mdx.md - raw/github_doc-docs-docs-supported-models-diffusion-language-models-mdx.md - raw/github_doc-docs-docs-supported-models-mindspore-models-mdx.md - raw/github_doc-docs-docs-supported-models-modelscope-mdx.md - raw/github_doc-docs-docs-supported-models-support-new-models-mdx.md - raw/github_doc-docs-docs-supported-models-transformers-fallback-mdx.md # Change Log ## 2026-08-24 — Initial build Built from the sgl-project/sglang docs (docs/docs/** — basic_usage, advanced_features, references, hardware, supported_models, developer_guide — plus the SGLang-Diffusion doc set) and 30 release mirrors. The large advanced-features (~41 files), hardware (~60 files incl. a ~50-file Ascend NPU subtree), and diffusion (29 files) sets were synthesized/grouped thematically, NOT one page per file. Content verified against `main` fetched 2026-08-24; newest release v0.5.18. **Pages (25):** 24 concepts + 1 summary (release digest). **Sourcing notes:** - Curation catch: the docs/docs/sglang/* set (29 files) is actually the **SGLang-Diffusion** subsystem (image/video generation), not core LLM architecture — it got its own two pages (diffusion-serving, diffusion-optimization). The core RadixAttention/architecture page was synthesized from README + hicache-design + scheduler/hyperparameter docs (no single dedicated architecture doc exists); the frontend DSL came from references/frontend. - Hardware synthesized to one page with representative sampling of the ~50-file Ascend NPU subtree (noted in-page). - A few advanced-features files were folded by theme (HiSparse→hierarchical-caching, R-Fork→lora-and-model-loading); forward-hooks/sglang-for-rl/vlm-query are minor and largely covered elsewhere or left out (noted). - Newest stable v0.5.18; the digest also tracks the separate gateway-vX.Y.Z release line. Most concept pages cite mutable `main` documentation fetched 2026-08-24, not a v0.5.18 tag/commit — treat page bodies as verified against that snapshot rather than pinned to the v0.5.18 release unless a page cites a release mirror directly (installation.md is the notable exception with genuinely v0.5.18-pinned examples). - Strong XL candidate later: the ~200-flag server-arguments reference, per-hardware deep tuning (esp. Ascend), and the diffusion acceleration stack are natural Pro-tier reference material. ## 2026-08-24 — XL (Pro) edition built Curated a 13-page XL tier from the already-gathered docs (no re-gather): the full server-arguments reference (435 flags / 38 sections) and environment-variable catalog (320 vars); the Model Gateway ops reference and benchmarking/profiling toolchain; deep references for parallelism/disaggregation, HiCache, speculative-decoding methods (EAGLE-2/3, MTP, DFLASH, STANDALONE, NGRAM + adaptive), and the quantization method×hardware matrix; and per-hardware deep tuning (NVIDIA, AMD ROCm, Intel XPU/CPU, Ascend NPU — the 50-file subtree synthesized with DeepSeek-R1 rendered in full and the other model tutorials characterized, TPU & others). Base index gained an "XL Edition (Pro)" section. ## 2026-08-24 — Fact-check remediation (base + XL) Claim-vs-source fact-check applied to both tiers (reviewer reports + independent per-page re-derivation). Base: fixed exact-value/operational errors (Ascend flag, spec-decode flag, mem-fraction default, cURL port, diffusion env vars, FP4 GSM8K numbers), architecture overstatements, six release-digest chronology errors; confidence->medium; version provenance clarified ("verified against main fetched 2026-08-24; newest release v0.5.18"). XL (13 pages): fixed the ModelOpt --quantization contradiction, the Ascend 1024-cap/DeepSeek-config/support-matrix errors, NVIDIA universal-support claims, Jetson/Apple/AMD/Intel overstatements, Mooncake shell-break, false-completeness claims, 7 dead links, env count (321/34), and attributed all first-party perf claims. A few reviewer findings were REJECTED on independent re-check (e.g. base total_tokens is real; XL admin-key markup was already valid). --- title: "SGLang Release Digest (v0.5.x line)" type: summary tags: [overview, advanced, well-established] created: 2026-08-24 updated: 2026-08-24 sources: ["raw/github_release-release-v0-5-1.md", "raw/github_release-release-v0-5-2.md", "raw/github_release-release-v0-5-3.md", "raw/github_release-release-v0-5-4.md", "raw/github_release-release-v0-5-5.md", "raw/github_release-release-v0-5-6.md", "raw/github_release-v0-5-7.md", "raw/github_release-v0-5-8.md", "raw/github_release-v0-5-9.md", "raw/github_release-v0-5-10rc0.md", "raw/github_release-v0-5-10.md", "raw/github_release-v0-5-10-post1.md", "raw/github_release-v0-5-11.md", "raw/github_release-v0-5-12.md", "raw/github_release-v0-5-12-post1.md", "raw/github_release-v0-5-13.md", "raw/github_release-v0-5-14.md", "raw/github_release-v0-5-15.md", "raw/github_release-v0-5-15-post1.md", "raw/github_release-v0-5-16.md", "raw/github_release-v0-5-17.md", "raw/github_release-v0-5-18.md", "raw/github_release-release-gateway-v0-1-9.md", "raw/github_release-release-gateway-v0-2-0.md", "raw/github_release-release-gateway-v0-2-1.md", "raw/github_release-release-gateway-v0-2-2.md", "raw/github_release-release-gateway-v0-2-3.md", "raw/github_release-release-gateway-v0-2-4.md", "raw/github_release-release-gateway-v0-3-0.md", "raw/github_release-release-gateway-v0-3-1.md"] confidence: medium sglang_version: "v0.5.18" --- ## Key Points **Cadence.** Core SGLang releases (`vX.Y.Z`) ship roughly every 2–4 weeks: v0.5.1 (2025-08-23) → v0.5.2 (09-12) → v0.5.3 (10-06) → v0.5.4 (10-26) → v0.5.5 (11-06) → v0.5.6 (12-03) → v0.5.7 (2026-01-01) → v0.5.8 (01-23) → v0.5.9 (02-24) → v0.5.10rc0 (03-28) → v0.5.10 (04-06) → v0.5.11 (05-05) → v0.5.12 (05-16) → v0.5.13 (06-13) → v0.5.14 (06-26) → v0.5.15 (07-10) → v0.5.16 (07-25) → v0.5.17 (08-08) → v0.5.18 (08-22, newest stable in the 2026-08-24 source snapshot, 710 PRs / 212 contributors). Point releases (`.postN`, e.g. v0.5.10.post1, v0.5.12.post1, v0.5.15.post1) land within days of their parent when a day-0 model or a major feature needs an urgent stability fix — v0.5.12.post1 alone cherry-picked 12 DeepSeek-V4 fixes onto the release branch just 10 days after v0.5.12. The separate **Model Gateway** (Rust router/proxy, formerly "SGLang Router") has its own `gateway-vX.Y.Z` tag line and release cadence, decoupled from core: v0.1.9 → v0.2.0 → v0.2.1 → v0.2.2 → v0.2.3 (all landed 2025-11-17, suggesting a batched backfill) → v0.2.4 (12-10) → v0.3.0 (12-24) → v0.3.1 (2026-01-09). **Day-0 model support is the recurring headline act.** Nearly every release leads with same-day support for a newly announced frontier model; in earlier releases these were announced with a linked blog post or model page (DeepSeek-V3.2 with sparse attention, v0.5.3; Kimi-K2-Thinking and MiniMax-M2, v0.5.5; DeepSeek-V3.2/V3.2-Speciale, v0.5.6; Mimo-V2-Flash, Nemotron-Nano-v3, and LLaDA-2.0, v0.5.7), while later releases (from roughly v0.5.13 onward) consistently pair the announcement with a `docs.sglang.io/cookbook` recipe: GLM-4.7-Flash (v0.5.8), Kimi-K2.5, Qwen3.5, MiniMax-2.5, GLM-5 (v0.5.9), Gemma 4, GLM-5.1, Qwen3.6, MiMo-V2.5, Kimi-K2.6 (v0.5.11), DeepSeek-V4 across NVIDIA B300/B200/H200/H100/GB200/GB300 and AMD MI35X on day 0 (v0.5.12, with a dedicated LMSYS blog and the DeepSeek-V4 cookbook), Nemotron-3-Ultra (v0.5.13), GLM-5.2, Kimi-K2.7-Code (v0.5.14), Hunyuan-3, GLM-5.2 NVFP4 production tuning (v0.5.15), Inkling (975B multimodal MoE, up to 71.7k tok/s input on Blackwell — release-note benchmark under that setup, not a cross-hardware guarantee) and DSpark speculative decoding (v0.5.16), Kimi-K3 (2.8T-param multimodal LatentMoE, native MXFP4, verified on GB300 and AMD MI35x) and MiniMax-H3 diffusion (v0.5.17), and Muse Glimmer, Intern-S2-Mobius, SANA-Video, LTX-2.5, and DeepSeek-V4-Pro-0813 (v0.5.18). **Speculative decoding evolved from a single fixed path to a pluggable, confidence-driven system.** Spec V2 (overlap-scheduled, tree drafting with topk>1) became the default in v0.5.11, matured across triton/FA3/MLA/aiter backends including `page_size>1` and Mamba/hybrid-linear models in v0.5.13, and Spec V1 was deprecated with EAGLE/MTP unified onto the V2 worker. DFLASH (a new high-throughput spec-decode kernel) shipped in v0.5.11 and expanded across backends and AMD ROCm through v0.5.13. **DSpark** — confidence-scheduled speculative decoding that sizes its verify window from the draft's own confidence instead of a fixed length — landed in v0.5.16 (383.7 tok/s at accept length ~5 on DeepSeek-V4-Pro TP8/B300, bs=1; release-note benchmark under that specific setup, not a cross-hardware guarantee), gained grammar-constrained decoding in v0.5.17, and gained logprobs plus MegaMoE support in v0.5.18. DFLASH is a distinct draft-verification algorithm, not a DSpark feature — it separately gained grammar-constrained decoding (v0.5.17), minimal support in the day-0 Inkling model (v0.5.17), and logprobs support (v0.5.18). **CUDA-graph coverage steadily expanded from decode-only to most of the model.** Piecewise CUDA graph became the default execution mode in v0.5.10; Piecewise/Breakable CUDA graph coverage was already extended to DSA models, Kimi-K2.5, and DeepSeek V4 in v0.5.13; breakable CUDA graph (BCG) became the default capture path in v0.5.15, with experimental full prefill CUDA graph landing the same release; v0.5.16 added (but did not yet default-enable) breakable prefill CUDA graph for DP attention, which became the default in v0.5.17; v0.5.17–18 further extended BCG/piecewise coverage (including to Kimi-K2.7 and MLA), plus made it runnable on AMD ROCm/HIP (v0.5.14) and diffusion DiTs (v0.5.16). **Parallelism and disaggregation kept adding new axes.** Elastic EP (partial GPU-failure tolerance for MoE, v0.5.10), decode context parallelism for MLA models (DeepSeek V3, Kimi K2 series, v0.5.15), a pluggable DCP comm-backend layer plus "q-replicate" (Helix) (v0.5.17), Waterfill and LPLB (linear-programming) MoE load-balancing (v0.5.14), CP-v2 zigzag strategy migration (v0.5.16), and a GPU staging buffer for PD disaggregation that cut RDMA request count on GQA models by ~1000x with ~5x TPS/GPU gains on Qwen3.5 with Prefill TP4+Decode DEP4 (v0.5.10; release-note benchmark under that specific setup, not a cross-hardware guarantee). A **native Rust server** replacing the Python front half of serving (tokenizer through GPU-scheduler handoff) shipped in v0.5.17, already including PD disaggregation support and prebuilt release artifacts as part of that same release. **Caching matured from a single radix tree to a tiered, session-aware system.** HiCache (hierarchical KV caching over pluggable storage backends) was announced in v0.5.2; HybridModel (SWA/Mamba) started launching HiCache via UnifiedTree by default in v0.5.13; UnifiedRadixTree became the default for SWA/Mamba/DSA models in v0.5.16; and a session-reference-aware Unified Radix Cache for agentic/RL-rollout multi-turn workloads (opt-in `--enable-session-radix-cache`, releasing references via `/close_session`) shipped in v0.5.17. **Quantization and hardware dependencies moved forward together.** CUDA moved to 13.0 with Torch 2.11 (v0.5.11), then Torch 2.13 with triton 3.7.1 (v0.5.18, also consolidating every compiled-kernel cache under `SGLANG_CACHE_DIR` — see [[concepts/references-and-faq]]). The experimental QServe (QoQ) W4A8 and FBGEMM FP8 quantization paths, plus the in-tree NVFP4 JIT kernels and `--fp4-gemm-backend cutlass`, were removed in v0.5.16 in favor of FlashInfer-backed NVFP4. NVFP4 MoE for DeepSeek-V4 landed in v0.5.14; a `quark_mxfp4` path that dequantizes ModelOpt/Quark NVFP4 checkpoints and requantizes to MXFP4 at load — letting NVFP4 checkpoints run on AMD GPUs — shipped in v0.5.18 (97.5–100.2% GSM8K recovery vs. the NVFP4 reference across five large models — MiniMax-M2.7, GLM-5.1, Kimi-K2.6, Qwen3.5-397B, and DeepSeek-R1; release-note benchmark, not a cross-hardware guarantee). A native MLX backend brought Apple Silicon support without CUDA in v0.5.10 (see [[concepts/supported-hardware]]); the kernel-library consolidation under RFC #29630 started with the `sgl-kernel` → `sglang-kernel` package rename in v0.5.10 and reached its "finale" migration phase completion in v0.5.17 (v0.5.16 was still an intermediate migration phase, not the completion); DGX Spark support shipped in v0.5.4. **SGLang-Diffusion (image/video generation) grew from an announcement into a wide-capability system across this line** (this framing is this KB's summary judgment, not a completeness claim any single release note makes). It was announced in v0.5.5; by v0.5.8 it had a diffusers backend, multi-LoRA, and a ComfyUI plugin; v0.5.9–v0.5.13 added parallel VAE decoding, token-level sequence sharding, Nunchaku/FP8 support, realtime OpenAI-style video streaming with msgpack frame transport, and progressive-resolution growing across FLUX/FLUX.2/Qwen-Image/Wan/Z-Image; day-0 diffusion model support ran across the whole v0.5.10–v0.5.18 span (LTX-2 in v0.5.10, Cosmos3 in v0.5.13, MiniMax-H3 in v0.5.17, SANA-Video and LTX-2.5 in v0.5.18); v0.5.14–v0.5.18 additionally added data-parallel serving, cross-node sequence parallelism, and breakable CUDA graph for DiTs (e.g. a reported 1.56x H200 two-stage end-to-end speedup for LTX-2 in v0.5.18 — a release-note benchmark under that specific setup, not a general guarantee). **The Model Gateway (Rust router) is a separate, fast-moving component.** It evolved from a simple PD/regular router (v0.1.x) into a full API gateway: gRPC data-plane mode with a Rust-native tokenizer/reasoning-parser/tool-parser stack, OpenAI-compatible Responses API for all models, multi-model Inference Gateway (IGW) mode, pluggable chat-history storage (memory/none/Oracle/PostgreSQL), mTLS and JWT/OIDC authentication, WASM middleware, and a 10–12x radix-tree cache-aware-routing performance overhaul with ~99% memory reduction per tree node (gateway v0.3.1, Jan 2026; release-note benchmark under its stated setup, not a cross-deployment guarantee). ## Relevant Concepts - [[concepts/sglang-overview]] - [[concepts/supported-hardware]] - [[concepts/supported-models]] - [[concepts/developer-and-benchmarking]] - [[concepts/references-and-faq]] - [[concepts/parallelism-and-disaggregation]] - [[concepts/hierarchical-caching]] - [[concepts/quantization]] - [[concepts/attention-backends-and-cuda-graph]] (planned) ## Source Metadata - **Type**: GitHub release notes (30 mirrors) — 22 core `sglang` releases (v0.5.1 through v0.5.18, including `.post1`/`rc0` point releases) and 8 `gateway-vX.Y.Z` Model Gateway releases. - **Author/organization**: sgl-project (SGLang), published via GitHub Releases. - **Date range**: 2025-08-23 (v0.5.1) through 2026-08-22 (v0.5.18, newest stable in the 2026-08-24 source snapshot). - **Identifiers**: `raw/github_release-release-v0-5-{1..6}.md`, `raw/github_release-v0-5-{7..18}.md` (with `-post1`/`rc0` variants), `raw/github_release-release-gateway-v0-{1.9,2.0,2.1,2.2,2.3,2.4,3.0,3.1}.md`.