wikis / SGLang / wiki / concepts / sampling-parameters.md view as markdown report a mistake

Definition

Sampling parameters are the set of controls that shape how SGLang's /generate endpoint turns model logits into output tokens — request framing (prompt/input format), core sampling knobs (temperature, top-p, top-k, min-p), repetition penalizers, constrained-decoding grammars, and miscellaneous generation options. The /generate endpoint is described as "the low-level endpoint of the runtime"; for a higher-level endpoint that automatically handles chat templates, the docs point to the OpenAI-compatible API instead (raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md).

How It Works

The /generate endpoint's request body is defined by GenerateReqInput (in io_struct.py); sampling_params within that body is itself defined by SamplingParams (in sampling_params.py). Both are defined in source code — the docs note "you can also read the source code to find more arguments," which is a pointer to incomplete documentation rather than a stated extensibility guarantee.

Request-level fields (outside sampling_params)

Sampling defaults source

By default (--sampling-defaults model, the default), SGLang initializes unset sampling parameters from the model's own generation_config.json. Pass --sampling-defaults openai at launch to use SGLang/OpenAI constant defaults instead. Any parameter can always be overridden per-request via sampling_params.

python -m sglang.launch_server --model-path <MODEL> --sampling-defaults model
python -m sglang.launch_server --model-path <MODEL> --sampling-defaults openai

Core parameters

Penalizers

Constrained decoding

Only one of json_schema / regex / ebnf may be set per request. SGLang supports two grammar backends: XGrammar (default; JSON schema, regex, and EBNF, using the GGML BNF format) and Outlines (JSON schema and regex only), selected at launch with --grammar-backend [xgrammar|outlines]. Note: the cited source is internally inconsistent about regex support — its backend-inventory section lists regex under both XGrammar and Outlines, but its worked example labels the regex case "(Outlines backend only)" and the EBNF case "(XGrammar backend only)." Verify current regex-backend support against the runtime/backend documentation rather than relying on either statement alone. Full detail is in structured outputs and tool calling.

Other options

Worked examples

Basic request:

import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {"temperature": 0, "max_new_tokens": 32},
    },
)

Multimodal request (image data alongside chat-formatted text):

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
                "<|im_start|>user\n<image>\nDescribe this image in a very short sentence.<|im_end|>\n"
                "<|im_start|>assistant\n",
        "image_data": "example_image.png",
        "sampling_params": {"temperature": 0, "max_new_tokens": 32},
    },
)

image_data accepts a file name, URL, or base64-encoded string.

Structured JSON output:

import json, requests

json_schema = json.dumps({
    "type": "object",
    "properties": {
        "name": {"type": "string", "pattern": "^[\\w]+$"},
        "population": {"type": "integer"},
    },
    "required": ["name", "population"],
})

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "Here is the information of the capital of France in the JSON format.\n",
        "sampling_params": {"temperature": 0, "max_new_tokens": 64, "json_schema": json_schema},
    },
)

Custom logit processor (requires launching with --enable-custom-logit-processor):

from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor

class DeterministicLogitProcessor(CustomLogitProcessor):
    def __call__(self, logits, custom_param_list):
        assert logits.shape[0] == len(custom_param_list)
        for i, param_dict in enumerate(custom_param_list):
            logits[i, :] = -float("inf")
            logits[i, param_dict["token_id"]] = 0.0
        return logits

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
        "sampling_params": {
            "temperature": 0.0,
            "max_new_tokens": 32,
            "custom_params": {"token_id": 5},
        },
    },
)

The same custom logit processor can be invoked via the OpenAI-compatible chat completions API using extra_body={"custom_logit_processor": ..., "custom_params": ...} — see server apis.

Key Parameters

See the full tables above. The parameters most commonly tuned per-request are temperature, top_p, top_k, max_new_tokens, stop/stop_token_ids, frequency_penalty/presence_penalty/repetition_penalty, and the constrained-decoding trio json_schema/regex/ebnf.

When To Use

• Use the request-level fields (text/input_ids/input_embeds, image_data, audio_data) to shape what goes into the model — text-only, multimodal, or precomputed-embedding input.
• Use core parameters (temperature, top_p, top_k, min_p) to trade off greedy vs. diverse output; temperature=0 for greedy sampling. Greedy sampling alone does not guarantee bit-identical/deterministic output across runs — dynamic batching and cache paths can still select different kernels; true determinism additionally requires --enable-deterministic-inference (see observability and determinism and references and faq).
• Use penalizers when a model repeats itself or over-uses certain tokens.
• Use constrained decoding (json_schema/regex/ebnf) when output must conform to a fixed structure — see structured outputs and tool calling.
• Use custom_logit_processor + custom_params for sampling logic not covered by the built-in parameters.
• Prefer --sampling-defaults openai when you want stable, well-known defaults across different models rather than each model's own generation_config.json values.

Risks & Pitfalls

• n > 1 (multiple outputs per request) is explicitly discouraged by the docs in favor of repeating the prompt across separate requests, for better control and efficiency.
• min_new_tokens forcing generation past a natural stop point "might lead to unintended behavior" when the output distribution is heavily skewed toward stop/EOS tokens.
• Only one of json_schema, regex, or ebnf can be set per request — combining them is not supported.
• The Outlines grammar backend does not support EBNF; the XGrammar backend does not use standard BNF (it uses "GGML BNF format") — grammar syntax is backend-specific.
• logprob_start_len defaults to -1 (output tokens only) — requesting logprobs over the prompt itself requires explicitly setting a different start offset.
• Sampling defaults are pulled from generation_config.json unless --sampling-defaults openai is set at launch — a value considered "unset" by the caller may still take on a per-model default silently.
• return_routed_experts requires the server to have been launched with --enable-return-routed-experts; requesting it without that flag will not work.

Related Concepts

• sglang overview
• sending requests
• server apis
• offline engine
• structured outputs and tool calling
• lora and model loading

Sources

• raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md