SamplingParams

The SamplingParams class controls how text is generated from language models. It configures sampling strategies, stopping conditions, penalties, and output options.

Constructor

from tensorrt_llm.sampling_params import SamplingParams

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    max_tokens=256
)

Parameters

Token Generation

int

default:"32"

Maximum number of tokens to generate per output sequence.

int

default:"None"

Minimum number of tokens to generate. Values < 1 have no effect. Prevents early stopping.

Sampling Strategy

float

default:"None"

Temperature for sampling (≥ 0). Controls randomness:

0.0: Greedy decoding (deterministic)
< 1.0: More focused, deterministic outputs
= 1.0: Standard sampling
> 1.0: More random, creative outputs

If None and neither top_p nor top_k are specified, defaults to greedy decoding.

float

default:"None"

Nucleus sampling threshold (0 to 1). Only tokens with cumulative probability ≥ top_p are considered.

1.0: Consider all tokens (standard sampling)
< 1.0: Consider only most probable tokens

If None and neither temperature nor top_k are specified, defaults to greedy decoding.

int

default:"None"

Sample from the top K most likely tokens.

0: Consider all tokens
1: Greedy decoding
> 1: Limit sampling to top K tokens

If None and neither temperature nor top_p are specified, defaults to greedy decoding.

float

default:"None"

Lower bound for top-P decay algorithm. Defaults to 1e-6.

float

default:"None"

Decay factor for top-P algorithm. Defaults to 1.0.

int

default:"None"

Token ID where top-P decay resets. Defaults to 1.

float

default:"None"

Minimum token probability threshold. Scales the most likely token to determine minimum probability. Defaults to 0.0.

int

default:"None"

Random seed for reproducible sampling. Defaults to 0.

Beam Search

bool

default:"False"

Enable beam search instead of sampling. When True, best_of becomes the beam width.

int

default:"1"

Number of output sequences to return per prompt.

int

default:"None"

Number of sequences to generate for selection:

Sampling mode: Generate best_of sequences, return top n by cumulative log probability
Beam search mode: Use beam width of best_of, return top n

Must satisfy best_of >= n. Defaults to n.

float

default:"None"

Diversity penalty for beam search. Values > 1.0 encourage diverse beams. Defaults to 1.0.

List[int]

default:"None"

Array of beam widths for variable-beam-width search.

float

default:"None"

Exponential penalty for sequence length in beam search. Defaults to 0.0.

int

default:"None"

Stop beam search when beam_width complete sequences are generated. Defaults to 1.

Stopping Conditions

int

default:"None"

End-of-sequence token ID. Generation stops when this token is generated. Defaults to tokenizer’s EOS token.

int

default:"None"

Padding token ID. Defaults to end_id.

str | List[str]

default:"None"

Stop string(s). Generation stops when any of these strings are generated.

SamplingParams(stop=["\n\n", "END", "###"])

List[int]

default:"None"

Stop token IDs. Generation stops when any of these tokens are generated.

bool

default:"False"

Include stop string in the output text. When False, stop strings are removed.

bool

default:"False"

Continue generation after EOS token is generated.

Bad Words / Tokens

str | List[str]

default:"None"

Bad string(s) to avoid. When these would be generated, they are redirected to alternative tokens.

List[int]

default:"None"

Token IDs to avoid during generation.

Repetition Control

float

default:"None"

Penalty for repeating tokens:

< 1.0: Encourage repetition
= 1.0: No penalty (default)
> 1.0: Discourage repetition

float

default:"None"

Penalty for tokens that have already appeared (independent of frequency):

< 0.0: Encourage repetition
= 0.0: No penalty (default)
> 0.0: Discourage repetition

float

default:"None"

Penalty based on token frequency in generated text:

< 0.0: Encourage repetition
= 0.0: No penalty (default)
> 0.0: Discourage repetition (stronger for more frequent tokens)

int

default:"None"

Number of prompt tokens to ignore for presence/frequency penalties. Defaults to 0.

int

default:"None"

Prevent repetition of n-grams of this size. Defaults to very large value (no limit).

Output Control

int

default:"None"

Number of log probabilities to return per output token:

None: No log probabilities
0: Only the sampled token’s log probability
K > 0: Top-K log probabilities plus sampled token (if not in top-K)

int

default:"None"

Number of log probabilities to return per prompt token. Same format as logprobs.

LogprobMode

default:"LogprobMode.RAW"

Log probability calculation mode:

LogprobMode.RAW: Raw log probabilities from model output
LogprobMode.PROCESSED: After applying sampling parameters (temperature, top-k, top-p)

bool

default:"False"

Return full logits tensor for prompt tokens.

bool

default:"False"

Return full logits tensor for generated tokens.

bool

default:"True"

Exclude input tokens from output token IDs.

bool

default:"False"

Return encoder hidden states for encoder-decoder models.

bool

default:"False"

Include performance metrics in output (TTFT, latency, throughput, etc.).

List[str]

default:"None"

Additional model outputs to gather (model-specific).

Tokenization

bool

default:"True"

Convert output token IDs to text.

bool

default:"True"

Add special tokens (BOS, EOS) when encoding the prompt.

bool

default:"True"

Skip special tokens when decoding output text.

bool

default:"True"

Add spaces between special tokens in decoded output.

int

default:"None"

Truncate prompt to last K tokens (left truncation). Must be ≥ 1.

Advanced

torch.Tensor

default:"None"

Embedding bias tensor of shape [vocab_size] with dtype float32.

LogitsProcessor | List[LogitsProcessor]

default:"None"

Custom logits processor callback(s) to modify logits before sampling. Can be a single processor or list.

bool

default:"False"

Apply batched logits processor. Processor must be provided when initializing LLM.

LookaheadDecodingConfig

default:"None"

Configuration for lookahead decoding optimization.

GuidedDecodingParams

default:"None"

Guided decoding parameters for structured output (JSON, regex, grammar).

Usage Examples

Basic Sampling

from tensorrt_llm.sampling_params import SamplingParams

# Balanced creativity
params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=256
)

Greedy Decoding

# Deterministic output
params = SamplingParams(
    temperature=0.0,  # or top_k=1
    max_tokens=100
)

Beam Search

params = SamplingParams(
    use_beam_search=True,
    best_of=4,  # beam width
    n=1,        # return best sequence
    max_tokens=200
)

Stop Strings

params = SamplingParams(
    stop=["\n\n", "###", "END"],
    include_stop_str_in_output=False,
    max_tokens=500
)

Multiple Outputs

# Generate 5 candidates, return best 3
params = SamplingParams(
    n=3,
    best_of=5,
    temperature=0.8,
    max_tokens=150
)

Log Probabilities

params = SamplingParams(
    logprobs=5,              # Top-5 token log probs
    prompt_logprobs=3,       # Top-3 for prompt tokens
    max_tokens=100
)

output = llm.generate("Hello", sampling_params=params)
for token_logprobs in output.outputs[0].logprobs:
    print(token_logprobs)  # Dict[token_id -> Logprob]

Repetition Control

params = SamplingParams(
    repetition_penalty=1.2,      # Discourage repetition
    frequency_penalty=0.5,       # Penalize frequent tokens
    presence_penalty=0.3,        # Penalize any repeated tokens
    no_repeat_ngram_size=3,      # No 3-gram repetition
    max_tokens=300
)

Structured Output with Guided Decoding

from tensorrt_llm.sampling_params import SamplingParams, GuidedDecodingParams

# JSON output
params = SamplingParams(
    guided_decoding=GuidedDecodingParams(
        json={
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "age": {"type": "integer"}
            },
            "required": ["name", "age"]
        }
    ),
    max_tokens=200
)

# Regex pattern
params = SamplingParams(
    guided_decoding=GuidedDecodingParams(
        regex=r"\d{3}-\d{3}-\d{4}"  # Phone number format
    ),
    max_tokens=50
)

Performance Monitoring

params = SamplingParams(
    return_perf_metrics=True,
    max_tokens=256
)

output = llm.generate("Test prompt", sampling_params=params)
metrics = output.outputs[0].request_perf_metrics
print(f"Time to first token: {metrics.ttft}")
print(f"Throughput: {metrics.throughput}")

GuidedDecodingParams

Parameters for structured output generation:

from tensorrt_llm.sampling_params import GuidedDecodingParams

guided = GuidedDecodingParams(
    json_object=True  # Any valid JSON object
)
# OR
guided = GuidedDecodingParams(
    json={...}  # Specific JSON schema
)
# OR
guided = GuidedDecodingParams(
    regex="pattern"  # Regex pattern
)
# OR
guided = GuidedDecodingParams(
    grammar="EBNF grammar"  # EBNF grammar
)

LogprobMode

Enum for log probability modes:

from tensorrt_llm.sampling_params import LogprobMode

LogprobMode.RAW        # Raw model output logits
LogprobMode.PROCESSED  # After temperature/top-k/top-p

Python API

CLI Tools

Configuration

SamplingParams

SamplingParams

Constructor

Parameters

Token Generation

Sampling Strategy

Beam Search

Stopping Conditions

Bad Words / Tokens

Repetition Control

Output Control

Tokenization

Advanced

Usage Examples

Basic Sampling

Greedy Decoding

Beam Search

Stop Strings

Multiple Outputs

Log Probabilities

Repetition Control

Structured Output with Guided Decoding

Performance Monitoring

GuidedDecodingParams

LogprobMode

​SamplingParams

​Constructor

​Parameters

​Token Generation

​Sampling Strategy

​Beam Search

​Stopping Conditions

​Bad Words / Tokens

​Repetition Control

​Output Control

​Tokenization

​Advanced

​Usage Examples

​Basic Sampling

​Greedy Decoding

​Beam Search

​Stop Strings

​Multiple Outputs

​Log Probabilities

​Repetition Control

​Structured Output with Guided Decoding

​Performance Monitoring

​Related Classes

​GuidedDecodingParams

​LogprobMode

SamplingParams

Constructor

Parameters

Token Generation

Sampling Strategy

Beam Search

Stopping Conditions

Bad Words / Tokens

Repetition Control

Output Control

Tokenization

Advanced

Usage Examples

Basic Sampling

Greedy Decoding

Beam Search

Stop Strings

Multiple Outputs

Log Probabilities

Repetition Control

Structured Output with Guided Decoding

Performance Monitoring

Related Classes

GuidedDecodingParams

LogprobMode