---
title: "SGLang Overview"
type: concept
tags: [overview, foundational, user]
created: 2026-08-24
updated: 2026-08-24
sources: ["raw/github_doc-readme-md.md", "raw/github_doc-docs-docs-get-started-quickstart-mdx.md", "raw/github_doc-docs-docs-basic-usage-overview-mdx.md"]
confidence: medium
sglang_version: "v0.5.18"
---

## Definition

SGLang is a high-performance serving framework for large language models (LLMs) and multimodal models. It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters. SGLang is open-source, hosted under the non-profit organization LMSYS. The upstream README itself claims SGLang "has become the de facto industry standard" for open-source LLM inference and reports deployments running on over 400,000 GPUs worldwide (raw/github_doc-readme-md.md, 2026-08-24 snapshot) — these are the project's own self-reported claims, not independently verified by this KB.

## How It Works

This KB frames SGLang's position as resting on three pillars — this is curator synthesis for organizing this wiki, not a "three pillars" architecture that the basic-usage overview doc itself states; that source is simply a flat list of links to the OpenAI/Anthropic/Ollama/offline-engine/native/sampling-parameters/model-usage docs (raw/github_doc-docs-docs-basic-usage-overview-mdx.md):

1. **A fast runtime.** The core engine provides efficient serving through RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, quantization (FP4/FP8/INT4/AWQ/GPTQ), and multi-LoRA batching (raw/github_doc-readme-md.md). See [[concepts/architecture-and-radixattention]] for the caching mechanics.
2. **A frontend DSL.** SGLang also ships a frontend DSL (see [[concepts/frontend-dsl]]); none of this page's three cited sources defines or substantively describes it (the README only links a "Frontend Tutorial" without elaborating), so treat any definition of it here as editorial framing rather than a sourced claim.
3. **An OpenAI-compatible server.** A served HTTP API surface — OpenAI-compatible endpoints, plus native, Anthropic-compatible, and Ollama-compatible APIs — so existing client tooling can talk to a self-hosted SGLang deployment. The "without changes" / drop-in claim is explicitly documented for the OpenAI-compatible API only (raw/github_doc-docs-docs-get-started-quickstart-mdx.md: "SGLang is fully OpenAI API-compatible"); the overview doc lists the Anthropic- and Ollama-compatible docs but does not itself state drop-in/no-changes behavior for them. See [[concepts/server-apis]] for the full API surface and [[concepts/offline-engine]] for in-process usage without a server.

A typical usage flow (raw/github_doc-docs-docs-get-started-quickstart-mdx.md): install the package, launch an inference server via `python3 -m sglang.launch_server`, then send requests using cURL, the OpenAI Python client, plain `requests`, or SGLang's native `/generate` endpoint. See [[concepts/installation]] and [[concepts/sending-requests]] for the mechanics of each step.

## Key Parameters

- **Broad model support** — language models (Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, Mistral, etc.), embedding models (e5-mistral, gte, mcdse), reward models (Skywork), and diffusion models (WAN, Qwen-Image); compatible with most Hugging Face models and OpenAI APIs (raw/github_doc-readme-md.md). See [[concepts/supported-models]].
- **Extensive hardware support** — NVIDIA GPUs (GB200/B300/H100/A100/Spark/5090), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, and more (raw/github_doc-readme-md.md). See [[concepts/supported-hardware]].
- **RL & post-training backbone** — used as a rollout backend for training frontier models, with native RL integrations and adoption by post-training frameworks such as AReaL, Miles, slime, Tunix, and verl (raw/github_doc-readme-md.md).

## When To Use

SGLang targets teams that need to self-host LLM/multimodal inference at low latency and high throughput, whether on a single GPU or a large distributed cluster, and that want drop-in, no-changes compatibility with OpenAI-style client code (raw/github_doc-docs-docs-get-started-quickstart-mdx.md) rather than building request/response plumbing from scratch. SGLang also ships Anthropic- and Ollama-compatible APIs (raw/github_doc-docs-docs-basic-usage-overview-mdx.md), but the "no changes" / drop-in framing is specifically documented for OpenAI compatibility — see [[concepts/server-apis]] for what's actually documented per API surface. It is also positioned as a production-grade rollout backend for RL/post-training pipelines (raw/github_doc-readme-md.md).

## Risks & Pitfalls

- The README's "News" section and feature list describe a fast-moving project (frequent day-0 model support announcements, active blog cadence); pinning a specific version (this KB targets v0.5.18) matters for reproducibility. Note that the underlying raw docs cited throughout this KB are mutable `main`-branch snapshots fetched 2026-08-24, not files pinned to the v0.5.18 tag — see [[concepts/installation]] for version-pinning guidance with Docker and source installs.
- SGLang layers several serving APIs (OpenAI, native, Anthropic, Ollama) on one server; picking the wrong one for a task (e.g., using the native `/generate` endpoint when an OpenAI-compatible client library is expected) adds unnecessary complexity — see [[concepts/server-apis]].

## Related Concepts

- [[concepts/installation]]
- [[concepts/sending-requests]]
- [[concepts/server-apis]]
- [[concepts/offline-engine]]
- [[concepts/sampling-parameters]]
- [[concepts/architecture-and-radixattention]]
- [[concepts/frontend-dsl]]
- [[concepts/supported-models]]
- [[concepts/supported-hardware]]

## Sources

- raw/github_doc-readme-md.md
- raw/github_doc-docs-docs-get-started-quickstart-mdx.md
- raw/github_doc-docs-docs-basic-usage-overview-mdx.md