Reinforcement Fine-Tuning (RFT) with OpenAI

Reinforcement Fine-Tuning (RFT) is a training technique that uses reward signals instead of labeled input-output pairs to improve a model’s reasoning and decision-making. Rather than teaching the model to reproduce a fixed “correct” answer, RFT teaches it to explore solution strategies, receive feedback from a grader, and reinforce the approaches that earn higher rewards. The result is a model that reasons more sharply in domains where correctness can be measured — making it well-suited for math, coding, scientific classification, and any task where you can define a clear grading function. This guide covers when to use RFT, how to design effective graders, and how to run and evaluate an RFT job end to end.

How RFT differs from SFT

Supervised fine-tuning (SFT) trains a model by minimizing the difference between predicted outputs and labeled targets. This works well when you have high-quality labeled data and the desired behavior can be expressed as an exact output. RFT takes a fundamentally different approach:

	SFT	RFT
Training signal	Labeled output pairs	Reward scores from a grader
What the model learns	To reproduce target outputs	To maximize reward through exploration
Data requirement	Correct answers for every example	Reward function; base model must have partial capability
Best for	Format, style, tone, distillation	Complex reasoning, verifiable correctness
Risk	Overfitting to examples	Reward hacking (optimizing the grader, not the task)

During RFT training, the model generates multiple candidate responses for each prompt, the grader evaluates each one, and the model’s weights are updated to favor responses that earned higher rewards. This iterative feedback loop can surface reasoning strategies that labeled data alone would not reveal.

RFT requires the base model to demonstrate at least partial capability on your task before training. If the model never produces a correct or near-correct output at baseline, there is no reward signal to learn from. Establish a baseline accuracy before committing to RFT.

When to use RFT

RFT is the right choice for tasks where:

Outputs are verifiable

The correctness of a response can be checked programmatically — a math answer, a diagnosis code, a SQL query that executes correctly, or a classification label.

Reasoning depth matters

The task requires multi-step reasoning rather than simple pattern matching. RFT helps models learn strategies, not just outputs.

Labels are scarce but feedback is easy

You may not have thousands of labeled examples, but you can define a reward function. RFT can make meaningful improvements with as few as 100 training examples.

Partial capability already exists

The base model succeeds on some examples but is inconsistent. RFT hill-climbs from that starting point.

RFT is not the right choice for tasks where the model has no underlying skill, for purely stylistic preferences without a correctness signal, or for tasks where it is impossible to define a reliable grader.

The grader: your reward function

The grader is the most important component of an RFT job. It defines what “good” means — and the model will optimize directly for whatever signal it receives. A poorly designed grader leads to reward hacking, where the model learns to score well according to the grader without actually improving on the real task. OpenAI’s RFT API supports several grader types:

Python graders (custom logic)

Python graders let you write arbitrary grading logic in a grade(sample, item) function. The function receives the model’s output and the training example, and returns a float between 0.0 and 1.0.

# Exact-match binary grader
def grade(sample: dict, item: dict) -> float:
    return 1.0 if sample["output_text"].strip() == item["reference_answer"].strip() else 0.0

You can also use fuzzy matching for tasks where multiple valid phrasings exist:

from rapidfuzz import fuzz, utils

def grade(sample: dict, item: dict) -> float:
    score = fuzz.token_set_ratio(
        sample["output_text"],
        item["reference_answer"],
        processor=utils.default_process
    )
    return score / 100.0

To submit a Python grader to the API, pass the source code as a string:

import inspect

def build_python_grader(grader_fn):
    source = inspect.getsource(grader_fn)
    # The API requires the function to be named `grade`
    source = source.replace(grader_fn.__name__, "grade", 1)
    return {"type": "python", "source": source}

Model graders (semantic scoring)

Model graders use a second LLM (such as GPT-4.1) to evaluate the model’s output semantically. This is useful when the task requires domain expertise to score, or when multiple correct phrasings exist.

GRADER_PROMPT = """
You are an expert grader. Compare the reference answer to the model's answer.
Respond with only a JSON object: {"result": <float 0.0-1.0>, "reasoning": "<one sentence>"}

Scoring rubric:
- Exact match: 1.0
- Clinical synonym or equivalent meaning: 0.7-0.9
- Same disease family or category: 0.5-0.7
- Partial overlap: 0.2-0.5
- Completely wrong: 0.0
"""

model_grader = {
    "type": "score_model",
    "name": "semantic_grader",
    "model": "gpt-4.1-2025-04-14",
    "input": [
        {"role": "system", "content": GRADER_PROMPT},
        {
            "role": "user",
            "content": "Reference: {{item.reference_answer}}\nModel answer: {{sample.output_text}}"
        }
    ],
    "pass_threshold": 0.75,
    "range": [0, 1],
    "sampling_params": {"temperature": 0, "seed": 42}
}

Before using a model grader in training, validate that its scores agree with human or expert judgments on a calibration set.

Multi graders (combined signals)

A multi grader combines several graders with a weighted formula. This lets you balance exact-match precision against semantic flexibility.

multi_grader = {
    "type": "multi",
    "graders": {
        "exact": {
            "name": "binary_grader",
            "image_tag": "2025-05-08",
            **build_python_grader(exact_match_grader),
        },
        "fuzzy": {
            "name": "fuzzy_grader",
            "image_tag": "2025-05-08",
            **build_python_grader(fuzzy_grader),
        },
    },
    "calculate_output": "0.15 * exact + 0.85 * fuzzy",
}

Avoiding reward hacking

Reward hacking occurs when the model finds ways to score well according to the grader without actually solving the underlying task. Common signs include:

Training reward improves rapidly while qualitative output quality stays flat or degrades
The model produces outputs that are superficially similar to references (high fuzzy score) but meaningless
The model learns idiosyncrasies of your specific grader rather than the general task

Guard against reward hacking by:

Evaluating on a held-out test set with a different (or stricter) grader
Using human review on a sample of high-scoring model outputs
Comparing model outputs before and after training qualitatively, not just by score

Example: math grader setup

Here is a worked example of an RFT grader for a math task, where the expected output is a numeric answer.

import re

def extract_number(text: str) -> float | None:
    """Extract the last number from model output."""
    matches = re.findall(r"-?\d+\.?\d*", text)
    if matches:
        return float(matches[-1])
    return None

def grade(sample: dict, item: dict) -> float:
    """
    Grade a math response.
    Returns 1.0 for exact numeric match, 0.5 for close answer, 0.0 otherwise.
    """
    predicted = extract_number(sample["output_text"])
    expected = float(item["reference_answer"])

    if predicted is None:
        return 0.0
    if predicted == expected:
        return 1.0
    if abs(predicted - expected) / (abs(expected) + 1e-9) < 0.01:
        # Within 1% relative error
        return 0.5
    return 0.0

The training data for this grader is a JSONL file where each line includes a messages list (the prompt) and a reference_answer field:

{"messages": [{"role": "user", "content": "What is 17 multiplied by 23?"}], "reference_answer": "391"}
{"messages": [{"role": "user", "content": "A rectangle has width 8 and height 5. What is its area?"}], "reference_answer": "40"}

Training configuration

Prepare and upload your dataset

Format your data as JSONL with messages and reference_answer fields. Upload it to the Files API:

from openai import OpenAI
client = OpenAI()

with open("rft_train.jsonl", "rb") as f:
    training_file = client.files.create(file=f, purpose="fine-tune")

print(training_file.id)

Create the RFT job

Pass your grader and the training file when creating the fine-tuning job. Specify "reinforcement" as the method type:

grader_payload = build_python_grader(grade)

job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="o4-mini-2025-04-16",
    method={
        "type": "reinforcement",
        "reinforcement": {
            "grader": grader_payload
        }
    }
)

print(job.id)

Monitor training progress

RFT jobs expose per-epoch reward metrics. Watch for a steadily increasing training reward alongside a stable or improving validation reward. If validation reward stagnates while training reward climbs, the model may be overfitting or reward hacking.

for event in client.fine_tuning.jobs.list_events(
    fine_tuning_job_id=job.id, limit=20
):
    print(event.message)

Evaluate the fine-tuned model

Once the job succeeds, retrieve the model name and run your evaluation suite against both the base model and the fine-tuned model to quantify the improvement.

job = client.fine_tuning.jobs.retrieve(job.id)
ft_model = job.fine_tuned_model

def benchmark(model_name, test_samples):
    correct = 0
    for sample in test_samples:
        response = client.chat.completions.create(
            model=model_name,
            messages=sample["messages"]
        )
        output = response.choices[0].message.content
        score = grade({"output_text": output}, sample)
        correct += score
    return correct / len(test_samples)

base_accuracy = benchmark("o4-mini", test_samples)
ft_accuracy = benchmark(ft_model, test_samples)

print(f"Base model accuracy: {base_accuracy:.2%}")
print(f"Fine-tuned accuracy: {ft_accuracy:.2%}")

Interpreting results

A successful RFT run should show:

Rising training reward across epochs, indicating the model is learning the reward signal
Stable or improving test reward, confirming that improvement generalizes
Qualitative improvement on held-out examples reviewed manually

RFT does not require thousands of samples. Because the model generates multiple candidate outputs per training example during training (trajectory sampling), even 100 well-chosen examples can produce meaningful improvement. Prioritize examples where the base model has partial capability — complete failures provide no gradient signal.

If training reward plateaus early, consider:

Revising your system prompt to give the model clearer guidance
Adjusting your grader to give more graduated (partial-credit) scores rather than hard binary signals
Switching from a string-match grader to a model grader for richer semantic feedback

Common pitfalls

Base model has no initial skill

RFT cannot teach a model a task it has never encountered. If baseline accuracy is near zero, the reward signal is too sparse for the model to learn from. Try prompting the base model with few-shot examples, or consider supervised fine-tuning first.

Grader is too strict or too lenient

A grader that returns only 0 or 1 for every example gives the model less to learn from than one that returns partial credit. Conversely, a grader that rewards almost everything provides no useful signal. Aim for a distribution of scores that correlates with actual quality.

Reward hacking

If training reward rises sharply but test reward does not follow, the model has found a shortcut specific to your grader. Review high-scoring model outputs manually and update the grader to close the loophole.

Overfitting to training examples

Like SFT, RFT can overfit. Use a validation set and monitor validation reward at each epoch. Consider reducing the number of training epochs if overfitting is observed.

Next steps

Fine-tuning (SFT and DPO)

If your task involves format, style, or tone rather than verifiable correctness, supervised fine-tuning or DPO is likely a better fit than RFT.

Evaluation

Learn how to build evaluation pipelines that measure model quality, detect hallucinations, and compare model versions — skills that apply directly to grader design for RFT.

Get Started

Models & Prompting

Agents & Automation

Embeddings & Search

Audio, Vision & Images

Fine-tuning & Evals

Reinforcement Fine-Tuning (RFT) with OpenAI

How RFT differs from SFT

When to use RFT

Outputs are verifiable

Reasoning depth matters

Labels are scarce but feedback is easy

Partial capability already exists

The grader: your reward function

Avoiding reward hacking

Example: math grader setup

Training configuration

Interpreting results

Common pitfalls

Next steps

Fine-tuning (SFT and DPO)

Evaluation

Build docs developers (and LLMs) love

Get Started

Models & Prompting

Agents & Automation

Embeddings & Search

Audio, Vision & Images

Fine-tuning & Evals

Documentation Index

​How RFT differs from SFT

​When to use RFT

Outputs are verifiable

Reasoning depth matters

Labels are scarce but feedback is easy

Partial capability already exists

​The grader: your reward function

​Avoiding reward hacking

​Example: math grader setup

​Training configuration

​Interpreting results

​Common pitfalls

​Next steps

Fine-tuning (SFT and DPO)

Evaluation

Build docs developers (and LLMs) love

How RFT differs from SFT

When to use RFT

The grader: your reward function

Avoiding reward hacking

Example: math grader setup

Training configuration

Interpreting results

Common pitfalls

Next steps