Vision and multimodal inputs with GPT-4o

GPT-4o and GPT-5.4 accept images alongside text in a single API call, enabling a wide range of visual understanding tasks: reading documents, captioning product photos, analyzing charts, inspecting video frames, and more. You pass images as either a publicly accessible URL or a base64-encoded string, and the model responds to your prompt in context with what it sees.

Send an image to GPT-4o

Include an image_url content block in your user message alongside the text prompt. The model treats both as part of the same turn.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/image.jpg"}
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)

Encode an image in base64

When you cannot expose a public URL — for example, images stored locally or behind authentication — encode the image as base64 and embed it directly in the request.

import base64
from openai import OpenAI

client = OpenAI()

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_data = encode_image("invoice.png")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Extract the total amount due from this invoice."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_data}"
                    }
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)

Supported MIME types include image/png, image/jpeg, image/gif, and image/webp.

Control image detail level

The detail parameter controls how much resolution the model uses when processing your image. Higher detail produces better results on dense content but costs more tokens.

auto (default)
low
high

The model decides the appropriate resolution based on image dimensions. Suitable for most general tasks.

{
    "type": "image_url",
    "image_url": {
        "url": "https://example.com/image.jpg",
        "detail": "auto"
    }
}

Processes the image as a 512×512 thumbnail regardless of original size. Faster and cheaper; use when fine details are not needed.

{
    "type": "image_url",
    "image_url": {
        "url": "https://example.com/image.jpg",
        "detail": "low"
    }
}

Tiles the image into multiple 512×512 segments and processes each one, then synthesizes the results. Use for dense scans, small text, engineering diagrams, or handwritten content.

{
    "type": "image_url",
    "image_url": {
        "url": "https://example.com/image.jpg",
        "detail": "high"
    }
}

When using the Responses API with GPT-5.4, the equivalent parameter is detail on input_image blocks, with values "auto" or "original". Use "original" for handwriting, low-quality scans, or images with very small labels.

Analyze multiple images

You can include any number of images in a single message. The model processes them all in context together.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Compare these two product photos and describe the differences."},
                {"type": "image_url", "image_url": {"url": "https://example.com/product-v1.jpg"}},
                {"type": "image_url", "image_url": {"url": "https://example.com/product-v2.jpg"}}
            ]
        }
    ]
)

Tag and caption images

A common production pattern is to send product images to the model and receive structured tags and captions for use in search indexes or content management systems.

import json
from openai import OpenAI

client = OpenAI()

def tag_and_caption(image_url: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a product tagging assistant. "
                    "Return a JSON object with keys: "
                    "'caption' (one descriptive sentence), "
                    "'tags' (list of 5-10 keywords)."
                )
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Tag and caption this product image."},
                    {"type": "image_url", "image_url": {"url": image_url}}
                ]
            }
        ],
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)

result = tag_and_caption("https://example.com/armchair.jpg")
print(result["caption"])
print(result["tags"])

Analyze video frames

GPT-4o does not accept raw video files, but you can extract frames using OpenCV and send them as a batch of images. With a large context window, you can cover an entire short video in one call.

Extract frames from the video

Use OpenCV to sample frames at a fixed interval.

import cv2
import base64

def extract_frames(video_path: str, max_frames: int = 50) -> list[str]:
    video = cv2.VideoCapture(video_path)
    total_frames = int(video.get(cv2.CAP_PROP_FRAME_COUNT))
    step = max(1, total_frames // max_frames)

    frames = []
    frame_index = 0
    while True:
        success, frame = video.read()
        if not success:
            break
        if frame_index % step == 0:
            _, buffer = cv2.imencode(".jpg", frame)
            encoded = base64.b64encode(buffer).decode("utf-8")
            frames.append(encoded)
        frame_index += 1

    video.release()
    return frames

Build the message content

Combine a text prompt with all the base64-encoded frames.

frames = extract_frames("wildlife.mp4", max_frames=30)

content = [{"type": "text", "text": "Describe what happens in this video, in sequence."}]
for frame in frames:
    content.append({
        "type": "image_url",
        "image_url": {
            "url": f"data:image/jpeg;base64,{frame}",
            "detail": "low"
        }
    })

Send to the model

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": content}]
)

print(response.choices[0].message.content)

Use detail: "low" for video frames to reduce token usage. The model can still follow action and motion at lower resolution for most narration or summarization tasks.

Document understanding and OCR

GPT-4o and GPT-5.4 excel at reading documents without requiring a separate OCR pipeline. Dense scans, handwritten forms, tables, and chart-heavy reports can all be interpreted and reasoned over in a single model pass.

Invoice extraction

Extract structured fields — totals, line items, dates, vendor names — from scanned invoices or receipts.

Form digitization

Transcribe handwritten or printed form data into JSON for downstream processing.

Chart and table reading

Describe trends, read axis values, and extract tabular data from screenshots or embedded figures.

Multi-page document QA

Answer questions about a document by sending multiple page images in one request.

Example: Extract fields from a scanned form

import base64
import json
from openai import OpenAI

client = OpenAI()

with open("form.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": (
                "Extract form fields from the image. "
                "Return a JSON object mapping field name to value. "
                "If a field is blank or illegible, use null."
            )
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_data}",
                        "detail": "high"
                    }
                }
            ]
        }
    ],
    response_format={"type": "json_object"}
)

fields = json.loads(response.choices[0].message.content)
print(fields)

For dense scans or pages with small handwritten text, use detail: "high" (or detail: "original" with GPT-5.4). Using "low" or "auto" on difficult document images can cause the model to miss details or misread characters.

GPT-5.4 and the Responses API

For workloads that require deep document reasoning, GPT-5.4 via the Responses API adds additional controls: a verbosity setting for faithful transcription and a reasoning.effort parameter that allocates more compute to multi-step visual tasks like charts and tables.

response = client.responses.create(
    model="gpt-5.4",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Transcribe this page verbatim, preserving layout."},
                {
                    "type": "input_image",
                    "image_url": f"data:image/png;base64,{image_data}",
                    "detail": "original"
                }
            ]
        }
    ],
    text={"verbosity": "high"},
    reasoning={"effort": "high"}
)

print(response.output_text)

Setting	When to use
`detail: "auto"`	General document QA and extraction on readable pages
`detail: "original"`	Dense scans, handwriting, tiny labels, or low-quality images
`verbosity: "high"`	Faithful transcription or markdown conversion
`reasoning.effort: "high"`	Charts, tables, diagrams requiring multi-step reasoning

Get Started

Models & Prompting

Agents & Automation

Embeddings & Search

Audio, Vision & Images

Fine-tuning & Evals

Vision and multimodal inputs with GPT-4o

Send an image to GPT-4o

Encode an image in base64

Control image detail level

Analyze multiple images

Tag and caption images

Analyze video frames

Document understanding and OCR

Invoice extraction

Form digitization

Chart and table reading

Multi-page document QA

Example: Extract fields from a scanned form

GPT-5.4 and the Responses API

Build docs developers (and LLMs) love

Get Started

Models & Prompting

Agents & Automation

Embeddings & Search

Audio, Vision & Images

Fine-tuning & Evals

Documentation Index

​Send an image to GPT-4o

​Encode an image in base64

​Control image detail level

​Analyze multiple images

​Tag and caption images

​Analyze video frames

​Document understanding and OCR

Invoice extraction

Form digitization

Chart and table reading

Multi-page document QA

​Example: Extract fields from a scanned form

​GPT-5.4 and the Responses API

Build docs developers (and LLMs) love

Send an image to GPT-4o

Encode an image in base64

Control image detail level

Analyze multiple images

Tag and caption images

Analyze video frames

Document understanding and OCR

Example: Extract fields from a scanned form

GPT-5.4 and the Responses API