wikis / llama.cpp / wiki / concepts / sampling-parameters.md view as markdown
Sampling Parameters
Definition
Sampling is how llama.cpp chooses the next token from the model's probability distribution over its vocabulary. Rather than always picking the single most likely token, sampling lets you trade off determinism against diversity. In llama.cpp this is done by applying a configurable chain of sampler stages, one after another, where each stage filters or reshapes the candidate token set before the final token is drawn.
How It Works
The model produces a raw score (a "logit") for every token in its vocabulary. The sampler chain transforms those logits/probabilities step by step in a fixed order, and the final stage picks a token.
The default sampler chain (set via --samplers) is:
penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature
The same order can be expressed in short form with --sampling-seq as
edskypmxt. You can reorder or remove stages to change behavior. Because the
order matters, moving temperature earlier or later, for example, changes the
result.
Brief description of each stage:
- temperature scales the logits; higher values make the distribution flatter and output more random.
- top-k keeps only the K most likely tokens.
- top-p (nucleus) keeps the smallest set of tokens whose probabilities sum to at least p.
- min-p keeps tokens whose probability is at least
min-p * (top token probability). - typical-p performs locally-typical sampling.
- top-n-sigma keeps tokens within n standard deviations of the distribution.
- repeat / frequency / presence penalties discourage repetition of tokens already seen.
- DRY ("Don't Repeat Yourself") applies an n-gram repetition penalty.
- XTC (exclude-top-choices) drops the most probable tokens with some probability to increase diversity.
- mirostat dynamically targets a perplexity setpoint.
Key Parameters
| Flag | Default | Notes |
|---|---|---|
--temp |
0.80 |
Logit scaling; higher = more random |
--top-k |
40 |
0 = off |
--top-p |
0.95 |
1.0 = off |
--min-p |
0.05 |
0.0 = off |
--typical / --typical-p |
1.00 |
Off |
--top-n-sigma / --top-nsigma |
-1.00 |
Off |
--xtc-probability |
0.00 |
|
--xtc-threshold |
0.10 |
|
--repeat-last-n |
64 |
Window for repetition penalty |
--repeat-penalty |
1.00 |
Off in CLI (see pitfall below) |
--presence-penalty |
0.00 |
|
--frequency-penalty |
0.00 |
|
--dry-multiplier |
0.00 |
DRY off when 0 |
--dry-base |
1.75 |
|
--dry-allowed-length |
2 |
|
--dry-penalty-last-n |
-1 |
Default breakers: \n : " * |
--mirostat |
0 |
1 = v1, 2 = v2 |
--mirostat-lr |
0.10 |
Learning rate |
--mirostat-ent |
5.00 |
Target entropy |
--dynatemp-range |
0.00 |
Dynamic temperature range |
--dynatemp-exp |
1.00 |
Dynamic temperature exponent |
-s / --seed |
-1 |
-1 = random |
There is also a recent adaptive-p sampler (PR #17927): --adaptive-target
(-1.00 = off) and --adaptive-decay (0.90). Additional controls include
-l / --logit-bias, --ignore-eos, and the experimental -bs /
--backend-sampling.
When To Use
- For more focused, deterministic output, lower
--tempand/or tighten--top-p/--top-k/--min-p. - For more creative or varied output, raise
--tempor enable XTC. - To suppress repetition or looping, enable DRY (
--dry-multiplier) or the repetition penalties. - To target a stable perplexity automatically, enable mirostat.
Risks & Pitfalls
- CLI vs server default mismatch: the server
/completionrequest defaultsrepeat_penaltyto1.1, while the CLI--repeat-penaltydefault is1.00(off). The same prompt can therefore behave differently between binary llama cli and the server unless you set the value explicitly. - Sampler order matters; reordering the chain changes output.
--backend-samplingis experimental.
Related Concepts
- server api — exposes these same samplers as request fields (and is where
the
repeat_penaltydefault differs). - gbnf grammars — constrains which tokens are allowed, complementary to sampling.
- kv cache and context — governs the context the sampler operates within.
- binary llama cli — the primary tool for setting these flags.
Sources
- cli-readme
- server-readme