Documentation Index
Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
Use this file to discover all available pages before exploring further.
The Generator class manages the text generation process. It handles the iterative generation of tokens and provides access to generated sequences.
Constructor
Generator(Model model, GeneratorParams generatorParams)
Creates a new generator instance.
The model to use for generation
Parameters controlling the generation process
using Microsoft.ML.OnnxRuntimeGenAI;
using Model model = new Model("/path/to/model");
using GeneratorParams generatorParams = new GeneratorParams(model);
using Generator generator = new Generator(model, generatorParams);
Generation Control Methods
GenerateNextToken()
Generates the next token in the sequence. This is the core method for the generation loop.
while (!generator.IsDone())
{
generator.GenerateNextToken();
}
IsDone()
Checks if generation is complete (end-of-sequence token generated or max length reached).
Returns: bool - true if generation is complete, false otherwise
if (generator.IsDone())
{
Console.WriteLine("Generation complete");
}
RewindTo(ulong newLength)
Rewinds the generator to a specific token length. Useful for chat scenarios where you want to keep the system prompt but remove the conversation history.
The token length to rewind to
// Save system prompt length
ulong systemPromptLength = generator.TokenCount();
// ... generate response ...
// Rewind to system prompt for next turn
generator.RewindTo(systemPromptLength);
inputIDs
ReadOnlySpan<int>
required
The token IDs to append
ReadOnlySpan<int> tokens = new int[] { 1, 2, 3, 4 };
generator.AppendTokens(tokens);
AppendTokenSequences(Sequences sequences)
Appends encoded sequences to the generator’s input.
The sequences to append (typically from Tokenizer.Encode())
using Sequences sequences = tokenizer.Encode("Hello, world!");
generator.AppendTokenSequences(sequences);
Tensor inputTensor = /* create tensor */;
generator.SetModelInput("custom_input", inputTensor);
Collection of named tensors to set as inputs
using NamedTensors inputs = processor.ProcessImagesAndAudios(prompt, images, audios);
generator.SetInputs(inputs);
Output Methods
GetNextTokens()
Returns the tokens generated in the last GenerateNextToken() call.
Returns: ReadOnlySpan<int> - The most recently generated tokens
generator.GenerateNextToken();
ReadOnlySpan<int> newTokens = generator.GetNextTokens();
GetSequence(ulong index)
Returns the complete token sequence for a specific sequence index.
The sequence index (0 for single sequence generation)
ReadOnlySpan<int> - The complete token sequence
// Get the complete output sequence
var outputSequence = generator.GetSequence(0);
string outputText = tokenizer.Decode(outputSequence);
TokenCount()
Returns the total number of tokens in the generator (including input and generated tokens).
Returns: ulong - The total token count
ulong totalTokens = generator.TokenCount();
Console.WriteLine($"Total tokens: {totalTokens}");
Tensor Access Methods
Retrieves an input tensor by name.
The name of the input tensor
Tensor - The input tensor
using Tensor inputTensor = generator.GetInput("input_ids");
GetOutput(string outputName)
Retrieves an output tensor by name.
The name of the output tensor
Tensor - The output tensor
using Tensor outputTensor = generator.GetOutput("logits");
Adapter Methods
SetActiveAdapter(Adapters adapters, string adapterName)
Activates a previously loaded adapter (for LoRA/fine-tuned models).
The name of the adapter to activate
using Adapters adapters = /* load adapters */;
generator.SetActiveAdapter(adapters, "my_adapter");
Complete Generation Example
Here’s a complete example of the generation loop:
examples/csharp/ModelChat/Program.cs:54-84
using Microsoft.ML.OnnxRuntimeGenAI;
using System.Diagnostics;
string modelPath = "/path/to/model";
// Load model
using Model model = new Model(modelPath);
using Tokenizer tokenizer = new Tokenizer(model);
// Prepare input
string prompt = "Once upon a time";
using var sequences = tokenizer.Encode(prompt);
// Create generator params
using GeneratorParams generatorParams = new GeneratorParams(model);
generatorParams.SetSearchOption("max_length", 200);
generatorParams.SetSearchOption("temperature", 0.7);
// Create generator and append input tokens
using Generator generator = new Generator(model, generatorParams);
generator.AppendTokenSequences(sequences);
// Run generation loop
var watch = Stopwatch.StartNew();
while (!generator.IsDone())
{
generator.GenerateNextToken();
}
watch.Stop();
// Get output and decode
var outputSequence = generator.GetSequence(0);
string outputString = tokenizer.Decode(outputSequence);
// Display results
Console.WriteLine("Output:");
Console.WriteLine(outputString);
var totalTokens = (int)generator.TokenCount();
var tokensPerSecond = totalTokens / watch.Elapsed.TotalSeconds;
Console.WriteLine($"Tokens: {totalTokens}, Time: {watch.Elapsed.TotalSeconds:0.00}s, Tokens/sec: {tokensPerSecond:0.00}");
Streaming Generation Example
examples/csharp/ModelChat/Program.cs:200-214
using Microsoft.ML.OnnxRuntimeGenAI;
using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);
using TokenizerStream tokenizerStream = tokenizer.CreateStream();
// ... setup generator ...
Console.Write("Output: ");
var watch = Stopwatch.StartNew();
while (!generator.IsDone())
{
generator.GenerateNextToken();
// Decode and print each token immediately
Console.Write(tokenizerStream.Decode(generator.GetNextTokens()[0]));
}
watch.Stop();
Console.WriteLine();
var totalTokens = (int)generator.TokenCount();
Console.WriteLine($"Tokens: {totalTokens}, Time: {watch.Elapsed.TotalSeconds:0.00}s");
Chat with Rewind Example
examples/csharp/ModelChat/Program.cs:314-382
using Microsoft.ML.OnnxRuntimeGenAI;
using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);
using TokenizerStream tokenizerStream = tokenizer.CreateStream();
// Create generator params
using GeneratorParams generatorParams = new GeneratorParams(model);
using Generator generator = new Generator(model, generatorParams);
// Encode system prompt
string systemPrompt = "You are a helpful AI assistant.";
var sequences = tokenizer.Encode(systemPrompt);
generator.AppendTokenSequences(sequences);
var systemPromptLength = (int)generator.TokenCount();
// Chat loop
while (true)
{
// Get user input
Console.Write("User: ");
string userPrompt = Console.ReadLine();
if (userPrompt == "quit()") break;
// Encode and append user prompt
sequences = tokenizer.Encode(userPrompt);
generator.AppendTokenSequences(sequences);
// Generate response
Console.Write("Assistant: ");
while (!generator.IsDone())
{
generator.GenerateNextToken();
Console.Write(tokenizerStream.Decode(generator.GetNextTokens()[0]));
}
Console.WriteLine();
// Rewind to system prompt for next turn
generator.RewindTo((ulong)systemPromptLength);
}
See Also