---
title: "Sampling Parameters"
type: concept
tags: [basic-usage, api, foundational, user]
created: 2026-08-24
updated: 2026-08-24
sources: ["raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md"]
confidence: medium
sglang_version: "v0.5.18"
---

## Definition

Sampling parameters are the set of controls that shape how SGLang's `/generate` endpoint turns model logits into output tokens — request framing (prompt/input format), core sampling knobs (temperature, top-p, top-k, min-p), repetition penalizers, constrained-decoding grammars, and miscellaneous generation options. The `/generate` endpoint is described as "the low-level endpoint of the runtime"; for a higher-level endpoint that automatically handles chat templates, the docs point to the [[concepts/server-apis|OpenAI-compatible API]] instead (raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md).

## How It Works

The `/generate` endpoint's request body is defined by `GenerateReqInput` (in `io_struct.py`); `sampling_params` within that body is itself defined by `SamplingParams` (in `sampling_params.py`). Both are defined in source code — the docs note "you can also read the source code to find more arguments," which is a pointer to incomplete documentation rather than a stated extensibility guarantee.

### Request-level fields (outside `sampling_params`)

| Argument | Type/Default | Description |
|---|---|---|
| `text` | `Optional[Union[List[str], str]] = None` | Input prompt(s). |
| `input_ids` | `Optional[Union[List[List[int]], List[int]]] = None` | Token IDs instead of text. |
| `input_embeds` | `Optional[Union[List[List[List[float]]], List[List[float]]]] = None` | Precomputed embeddings for input. |
| `image_data` | `Optional[...] = None` | Raw image (PIL/path/URL/base64), processor output, or precomputed embedding — single image, list, or list of lists. |
| `audio_data` | `Optional[Union[List[AudioDataItem], AudioDataItem]] = None` | Audio file name, URL, or base64 string. |
| `sampling_params` | `Optional[Union[List[Dict], Dict]] = None` | The sampling controls detailed below. |
| `rid` | `Optional[Union[List[str], str]] = None` | Request ID. |
| `return_logprob` | `Optional[Union[List[bool], bool]] = None` | Return token log-probabilities. |
| `logprob_start_len` | `Optional[Union[List[int], int]] = None` | Prompt offset from which to return logprobs; default `-1` = output tokens only. |
| `top_logprobs_num` | `Optional[Union[List[int], int]] = None` | Number of top logprobs per position. |
| `token_ids_logprob` | `Optional[Union[List[List[int]], List[int]]] = None` | Specific token IDs to return logprobs for. |
| `return_text_in_logprobs` | `bool = False` | Detokenize tokens in returned logprobs. |
| `stream` | `bool = False` | Stream output. |
| `lora_path` | `Optional[Union[List[Optional[str]], Optional[str]]] = None` | Path to a LoRA adapter. |
| `custom_logit_processor` | `Optional[Union[List[Optional[str]], str]] = None` | Serialized `CustomLogitProcessor` (via `.to_str()`). |
| `return_hidden_states` | `Union[List[bool], bool] = False` | Return hidden states. |
| `return_routed_experts` | `bool = False` | Return MoE routing data; requires `--enable-return-routed-experts`. Base64-encoded int32, shape `[num_tokens, num_layers, top_k]`; default returns the full sequence `[0, seqlen-1)` (RL workflows need the full sequence). |
| `routed_experts_start_len` | `int = 0` | Absolute start offset for `return_routed_experts`; must be in `[0, prompt_tokens]`. Useful in multi-turn RL rollouts to skip re-transferring already-collected routing for earlier turns. |

### Sampling defaults source

By default (`--sampling-defaults model`, the default), SGLang initializes unset sampling parameters from the model's own `generation_config.json`. Pass `--sampling-defaults openai` at launch to use SGLang/OpenAI constant defaults instead. Any parameter can always be overridden per-request via `sampling_params`.

```bash
python -m sglang.launch_server --model-path <MODEL> --sampling-defaults model
python -m sglang.launch_server --model-path <MODEL> --sampling-defaults openai
```

### Core parameters

| Argument | Type/Default | Description |
|---|---|---|
| `max_new_tokens` | `int = 128` | Max output length in tokens. |
| `stop` | `Optional[Union[str, List[str]]] = None` | Stop word(s); generation halts if sampled. |
| `stop_token_ids` | `Optional[List[int]] = None` | Stop word(s) as token IDs. |
| `stop_regex` | `Optional[Union[str, List[str]]] = None` | Stop on regex match. |
| `temperature` | `float` (model default; fallback `1.0`) | `0` = greedy sampling (does not by itself guarantee deterministic/reproducible output — see Risks & Pitfalls); higher = more diversity. |
| `top_p` | `float` (model default; fallback `1.0`) | Samples from the smallest token set whose cumulative probability exceeds `top_p`; `1` = unrestricted. |
| `top_k` | `int` (model default; fallback `-1`) | Randomly selects from the `k` highest-probability tokens. |
| `min_p` | `float` (model default; fallback `0.0`) | Samples from tokens with probability `> min_p * highest_token_probability`. |

### Penalizers

| Argument | Type/Default | Description |
|---|---|---|
| `frequency_penalty` | `float = 0.0` | Range `[-2, 2]`; penalizes tokens by frequency-so-far. Scaling grows linearly per occurrence. Negative encourages repetition, positive discourages it. |
| `presence_penalty` | `float = 0.0` | Range `[-2, 2]`; penalizes tokens that have appeared at all so far. Scaling is constant once a token has occurred (unlike frequency_penalty). |
| `repetition_penalty` | `float = 1.0` | Range `(0, 2]`; scales logits of previously generated tokens — `>1` discourages, `<1` encourages repetition; `1.0` is a no-op. |
| `min_new_tokens` | `int = 0` | Forces at least this many tokens before a stop word/EOS can end generation; can misbehave if the token distribution is heavily skewed toward stop tokens. |

### Constrained decoding

| Argument | Type/Default | Description |
|---|---|---|
| `json_schema` | `Optional[str] = None` | JSON schema constraint. |
| `regex` | `Optional[str] = None` | Regex constraint. |
| `ebnf` | `Optional[str] = None` | EBNF grammar constraint. |
| `structural_tag` | `Optional[str] = None` | Structural tag constraint. |

Only one of `json_schema` / `regex` / `ebnf` may be set per request. SGLang supports two grammar backends: **XGrammar** (default; JSON schema, regex, and EBNF, using the GGML BNF format) and **Outlines** (JSON schema and regex only), selected at launch with `--grammar-backend [xgrammar|outlines]`. Note: the cited source is internally inconsistent about regex support — its backend-inventory section lists regex under both XGrammar and Outlines, but its worked example labels the regex case "(Outlines backend only)" and the EBNF case "(XGrammar backend only)." Verify current regex-backend support against the runtime/backend documentation rather than relying on either statement alone. Full detail is in [[concepts/structured-outputs-and-tool-calling]].

### Other options

| Argument | Type/Default | Description |
|---|---|---|
| `n` | `int = 1` | Number of output sequences per request. Generating multiple outputs via `n > 1` is discouraged; repeating the prompt in separate requests offers better control and efficiency. |
| `ignore_eos` | `bool = False` | Don't stop generation on EOS. |
| `skip_special_tokens` | `bool = True` | Remove special tokens during decoding. |
| `spaces_between_special_tokens` | `bool = True` | Add spaces between special tokens during detokenization. |
| `no_stop_trim` | `bool = False` | Don't trim stop words/EOS from generated text. |
| `custom_params` | `Optional[List[Optional[Dict[str, Any]]]] = None` | Parameters passed through to a `CustomLogitProcessor`. Note: the cited source's own type table (list-of-dicts) and its worked example (a single dict, not a list) disagree on the exact shape — treat this as unresolved rather than certain. |

### Worked examples

**Basic request:**

```python
import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {"temperature": 0, "max_new_tokens": 32},
    },
)
```

**Multimodal request** (image data alongside chat-formatted text):

```python
response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
                "<|im_start|>user\n<image>\nDescribe this image in a very short sentence.<|im_end|>\n"
                "<|im_start|>assistant\n",
        "image_data": "example_image.png",
        "sampling_params": {"temperature": 0, "max_new_tokens": 32},
    },
)
```

`image_data` accepts a file name, URL, or base64-encoded string.

**Structured JSON output:**

```python
import json, requests

json_schema = json.dumps({
    "type": "object",
    "properties": {
        "name": {"type": "string", "pattern": "^[\\w]+$"},
        "population": {"type": "integer"},
    },
    "required": ["name", "population"],
})

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "Here is the information of the capital of France in the JSON format.\n",
        "sampling_params": {"temperature": 0, "max_new_tokens": 64, "json_schema": json_schema},
    },
)
```

**Custom logit processor** (requires launching with `--enable-custom-logit-processor`):

```python
from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor

class DeterministicLogitProcessor(CustomLogitProcessor):
    def __call__(self, logits, custom_param_list):
        assert logits.shape[0] == len(custom_param_list)
        for i, param_dict in enumerate(custom_param_list):
            logits[i, :] = -float("inf")
            logits[i, param_dict["token_id"]] = 0.0
        return logits
```

```python
response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
        "sampling_params": {
            "temperature": 0.0,
            "max_new_tokens": 32,
            "custom_params": {"token_id": 5},
        },
    },
)
```

The same custom logit processor can be invoked via the OpenAI-compatible chat completions API using `extra_body={"custom_logit_processor": ..., "custom_params": ...}` — see [[concepts/server-apis]].

## Key Parameters

See the full tables above. The parameters most commonly tuned per-request are `temperature`, `top_p`, `top_k`, `max_new_tokens`, `stop`/`stop_token_ids`, `frequency_penalty`/`presence_penalty`/`repetition_penalty`, and the constrained-decoding trio `json_schema`/`regex`/`ebnf`.

## When To Use

- Use the request-level fields (`text`/`input_ids`/`input_embeds`, `image_data`, `audio_data`) to shape what goes into the model — text-only, multimodal, or precomputed-embedding input.
- Use core parameters (`temperature`, `top_p`, `top_k`, `min_p`) to trade off greedy vs. diverse output; `temperature=0` for greedy sampling. Greedy sampling alone does not guarantee bit-identical/deterministic output across runs — dynamic batching and cache paths can still select different kernels; true determinism additionally requires `--enable-deterministic-inference` (see [[concepts/observability-and-determinism]] and [[concepts/references-and-faq]]).
- Use penalizers when a model repeats itself or over-uses certain tokens.
- Use constrained decoding (`json_schema`/`regex`/`ebnf`) when output must conform to a fixed structure — see [[concepts/structured-outputs-and-tool-calling]].
- Use `custom_logit_processor` + `custom_params` for sampling logic not covered by the built-in parameters.
- Prefer `--sampling-defaults openai` when you want stable, well-known defaults across different models rather than each model's own `generation_config.json` values.

## Risks & Pitfalls

- `n > 1` (multiple outputs per request) is explicitly discouraged by the docs in favor of repeating the prompt across separate requests, for better control and efficiency.
- `min_new_tokens` forcing generation past a natural stop point "might lead to unintended behavior" when the output distribution is heavily skewed toward stop/EOS tokens.
- Only one of `json_schema`, `regex`, or `ebnf` can be set per request — combining them is not supported.
- The Outlines grammar backend does not support EBNF; the XGrammar backend does not use standard BNF (it uses "GGML BNF format") — grammar syntax is backend-specific.
- `logprob_start_len` defaults to `-1` (output tokens only) — requesting logprobs over the prompt itself requires explicitly setting a different start offset.
- Sampling defaults are pulled from `generation_config.json` unless `--sampling-defaults openai` is set at launch — a value considered "unset" by the caller may still take on a per-model default silently.
- `return_routed_experts` requires the server to have been launched with `--enable-return-routed-experts`; requesting it without that flag will not work.

## Related Concepts

- [[concepts/sglang-overview]]
- [[concepts/sending-requests]]
- [[concepts/server-apis]]
- [[concepts/offline-engine]]
- [[concepts/structured-outputs-and-tool-calling]]
- [[concepts/lora-and-model-loading]]

## Sources

- raw/github_doc-docs-docs-basic-usage-sampling-params-mdx.md