Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/openai/openai-cookbook/llms.txt

Use this file to discover all available pages before exploring further.

OpenAI provides a complete audio stack: Whisper for transcribing recorded audio in over 50 languages, a text-to-speech (TTS) endpoint for generating natural-sounding voice output, and a Realtime API for building low-latency voice applications with streaming input and output. Whether you need to process uploaded meeting recordings, add a voice interface to a chatbot, or build live caption software, the same OpenAI client handles all three.

​
Transcribe audio with Whisper

The audio.transcriptions.create endpoint accepts an audio file and returns the transcribed text. It works well for voicemails, meeting recordings, podcasts, and any other pre-recorded content. 
from openai import OpenAI

client = OpenAI()

with open("audio.mp3", "rb") as audio_file:
    transcription = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file
    )

print(transcription.text)

​
Supported formats

Whisper accepts the following audio formats: mp3, mp4, mpeg, mpga, m4a, wav, and webm. The maximum file size per request is 25 MB. For longer recordings, split the audio into segments before sending.
Use a library like PyDub to trim silence from the start and end of recordings and to split long audio files into manageable segments before transcription.

​
Detect language automatically

By default, Whisper detects the spoken language automatically. You can also specify a language explicitly using an ISO-639-1 code to improve accuracy and skip the detection step. 
with open("recording.wav", "rb") as audio_file:
    transcription = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        language="fr"  # French
    )

​
Return timestamps

Add timestamp_granularities to receive word-level or segment-level timing alongside the transcript. Set response_format to "verbose_json" to access the full output structure. 
with open("audio.mp3", "rb") as audio_file:
    transcription = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        response_format="verbose_json",
        timestamp_granularities=["word"]
    )

for word in transcription.words:
    print(f"{word.word:20s}  {word.start:.2f}s – {word.end:.2f}s")

​
Improve accuracy with prompting

Whisper’s prompt parameter accepts a short text snippet (up to 224 tokens) that the model uses to calibrate its output style. Unlike GPT prompting, Whisper does not follow instructions — it imitates the style and vocabulary of the prompt text. There are two practical techniques:
Provide a comma-separated list of proper nouns, product names, or technical terms that Whisper might otherwise misspell. The model learns the correct spellings from context.
prompt = (
    "Acme Corp, Q3 earnings call. "
    "Products mentioned: NovaSpark Pro, DataBridge 2.0, CloudSync."
)

with open("earnings_call.wav", "rb") as f:
    transcription = client.audio.transcriptions.create(
        model="whisper-1",
        file=f,
        prompt=prompt
    )
The prompt is limited to 224 tokens. If you provide more, only the final 224 tokens are used. The prompt influences style and vocabulary — it does not allow you to issue instructions like “format as bullet points.”

​
Stitch multi-segment transcriptions

When you split a long recording into segments, pass the previous segment’s transcript as the prompt for the next segment. This maintains consistent vocabulary, speaker names, and punctuation across the full recording. 
segments = ["part_01.mp3", "part_02.mp3", "part_03.mp3"]
full_transcript = ""

for path in segments:
    with open(path, "rb") as f:
        result = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            prompt=full_transcript[-500:] or None  # last 500 chars as context
        )
    full_transcript += " " + result.text

print(full_transcript.strip())

​
Generate speech with TTS

The audio.speech.create endpoint converts text to spoken audio. Choose from six built-in voices and two model tiers depending on your quality and latency needs. 
response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="Hello! How can I help you today?"
)

response.stream_to_file("output.mp3")

​
Available voices

alloy

Neutral and balanced. A good default for general applications.

echo

Male, slightly more formal. Works well for instructional content.

fable

Warm and expressive. Suited for storytelling or creative content.

onyx

Deep and authoritative. Strong for announcements or narration.

nova

Bright and energetic. A good choice for assistants and chatbots.

shimmer

Soft and calm. Well suited for meditation or accessibility use cases.

​
TTS model tiers

ModelDescription
tts-1Optimized for low latency. Best for real-time applications.
tts-1-hdOptimized for audio quality. Best for recorded or published content.

​
Stream audio to a file or speaker

For longer inputs, stream audio as it is generated rather than waiting for the full response. 
with client.audio.speech.with_streaming_response.create(
    model="tts-1",
    voice="nova",
    input="This is a longer passage that benefits from streaming output.",
) as response:
    response.stream_to_file("streamed_output.mp3")

​
Supported output formats

The TTS endpoint defaults to MP3. Pass response_format to choose a different container. 
response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="Output as WAV for downstream audio processing.",
    response_format="wav"  # mp3 | opus | aac | flac | wav | pcm
)

​
Transcription methods compared

Different scenarios call for different approaches. The table below summarizes the key trade-offs.
MethodFirst token latencyBest forKey limitations
File upload, non-streamingSecondsVoicemail, meeting recordingsNo partial results; 25 MB max per request
File upload, streamingSub-second feelVoice memos, mobile appsStill requires a completed file before sending
Realtime WebSocketSub-secondLive captions, voice assistantsAudio must be PCM16, G711 ulaw, or G711 alaw; sessions limited to 30 min
Agents SDK VoicePipelineSub-secondAgentic voice workflowsPython-only beta; API surface may change

​
Real-time voice with the Realtime API

The Realtime API accepts a continuous audio stream over a WebSocket and returns transcription events as speech is detected. This is the right choice for live captioning, voice interfaces, and any application where users expect an immediate response to their voice.
1

Connect to the Realtime API

Open a WebSocket connection using your API key.
import asyncio
import websockets
import json
import os

async def connect_realtime():
    url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"
    headers = {
        "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
        "OpenAI-Beta": "realtime=v1"
    }
    async with websockets.connect(url, additional_headers=headers) as ws:
        print("Connected to Realtime API")
        # Send and receive events here
2

Send audio chunks

Append raw PCM16 audio data to the input audio buffer as it arrives.
import base64

async def send_audio(ws, pcm_chunk: bytes):
    event = {
        "type": "input_audio_buffer.append",
        "audio": base64.b64encode(pcm_chunk).decode("utf-8")
    }
    await ws.send(json.dumps(event))
3

Receive transcription events

Listen for conversation.item.input_audio_transcription.completed events to receive the final transcript for each turn.
async def listen(ws):
    async for message in ws:
        event = json.loads(message)
        if event["type"] == "conversation.item.input_audio_transcription.completed":
            print("Transcript:", event["transcript"])
The Realtime API requires audio in PCM16 format (24 kHz, mono, 16-bit signed). If your audio source produces a different format, use a library such as resampy or sounddevice to convert before sending.

​
Pre- and post-processing tips

Raw transcriptions sometimes need cleaning before they are useful in production. Common post-processing steps include:
  • Add punctuation: Pass the transcript through a GPT model with a prompt to insert missing punctuation and capitalize sentence starts.
  • Normalize numbers: Convert spoken numbers (“five two nine”) to digit form (“529”) using regex rules or a language model pass.
  • Handle Unicode: Normalize Unicode characters to remove unexpected encoding artifacts, especially from recordings with accented speech.
def clean_transcript(raw: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a transcript editor. "
                    "Add correct punctuation and capitalization. "
                    "Convert spoken numbers to digits. "
                    "Return only the corrected transcript."
                )
            },
            {"role": "user", "content": raw}
        ]
    )
    return response.choices[0].message.content

Build docs developers (and LLMs) love

Get started for freeTalk to us