Speech Synthesis and Recognition

Last updated July 24, 2026

AI Gateway offers models for speech synthesis and speech recognition. You can use them to convert text into spoken audio, add voice responses to your applications, transcribe calls, and turn other audio recordings into text.

For speech synthesis, or TTS (Text-to-Speech), AI Gateway provides the openai/gpt-4o-mini-tts model.

For speech recognition, or STT (Speech-to-Text), you can use openai/gpt-4o-mini-transcribe and openai/gpt-4o-transcribe.

Speech Synthesis Capabilities

The openai/gpt-4o-mini-tts model generates audio from text you provide. It supports different languages and lets you control how the speech sounds using an instruction—for example, setting the pace, intonation, emotional tone, or overall style.

You can choose from the following voices:

alloy
ash
ballad
coral
echo
fable
nova
onyx
sage
shimmer
verse
marin
cedar

You can listen to samples of each voice on OpenAI.fm.

The following audio formats are supported:

mp3: the default format
opus: suited for streaming and voice communication
aac: a compressed format commonly used on mobile devices
flac: lossless compression
wav: uncompressed audio
pcm: raw audio data with no file header

The wav and pcm formats work well for use cases where low latency matters. AI Gateway also supports streaming, so playback can start before the entire audio file has finished generating.

Transcription Capabilities

The openai/gpt-4o-mini-transcribe and openai/gpt-4o-transcribe models convert speech from an audio file into text. You can either let the model detect the language automatically or specify it explicitly in the request.

Transcription supports the following file formats: mp3, mp4, mpeg, mpga, m4a, wav, and webm. You can get the result back as JSON or as plain text.

You can also pass a prompt parameter with additional context in your request. This helps the model correctly recognize:

first and last names
product and company names
abbreviations
professional or technical terms
words that are hard to identify from pronunciation alone

If a recording is split into several chunks, you can pass the text of the previous chunk in the prompt parameter. This helps the model keep context between requests.

How Tokens Are Counted

Audio models count both text tokens and audio tokens. Text tokens depend on the amount of text involved, while audio tokens depend on the length of the audio.

For speech synthesis (TTS models):

input_tokens: text tokens from the input text
output_tokens: audio tokens from the generated recording. One second of audio counts as one audio token.

For example, if the model generates four seconds of audio, that's four output audio tokens. The number of input text tokens depends on the length of the source text.

For transcription (STT models):

input_tokens: audio tokens from the source recording, roughly 10 tokens per second
output_tokens: text tokens from the transcript

For example, a 10-second recording uses roughly 100 input audio tokens. The number of output text tokens depends on the length of the transcript.

Limitations And Notes

Audio models are only available through AI Gateway.
Streaming generation and playback are only supported for TTS.
Real-time transcription from a microphone isn't supported.
Transcription responses are available in JSON or plain text format.

Getting Started

To use the audio models, you'll need to create an AI Gateway API key.

The examples below use an environment variable called HOSTMAN_AI_TOKEN so you don't have to paste your key directly into your code. On Linux and macOS, you can set it with:

export HOSTMAN_AI_TOKEN="your_API_key"

Install the OpenAI library:

pip install openai

Create a client and point it to the AI Gateway base URL:

import os

from openai import OpenAI

client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)

Speech Synthesis

Speech synthesis uses the audio.speech.create() method.

Generating an Audio File

In the example below, the model converts text to speech and saves the result to a file called speech.mp3:

import os
from pathlib import Path

from openai import OpenAI

client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)

speech_file = Path("speech.mp3")

with client.audio.speech.with_streaming_response.create(
   model="openai/gpt-4o-mini-tts",
   voice="shimmer",
   input="Hello! This is a speech synthesis test.",
   instructions="Speak in a calm and friendly tone.",
   response_format="mp3",
) as response:
   response.stream_to_file(speech_file)

Parameters:

model: the speech synthesis model to use
voice: the voice used to read the text
input: the text to convert to speech
instructions: an instruction describing the speaking style
response_format: the format of the generated audio

Streaming Playback

To start playback before generation finishes, use the async client together with LocalAudioPlayer.

To use LocalAudioPlayer, install the numpy and sounddevice libraries:

pip install numpy sounddevice

Example of streaming playback:

import asyncio
import os

from openai import AsyncOpenAI
from openai.helpers import LocalAudioPlayer

client = AsyncOpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)

async def main() -> None:
   async with client.audio.speech.with_streaming_response.create(
       model="openai/gpt-4o-mini-tts",
       voice="shimmer",
       input="Hello! This is a streaming speech synthesis test.",
       response_format="pcm",
   ) as response:
       await LocalAudioPlayer().play(response)

if __name__ == "__main__":
   asyncio.run(main())

This example uses the pcm format, which lets you play the audio as it's received from the model.

Audio Transcription

Transcription uses the audio.transcriptions.create() method.

Transcribing a File

In the example below, the model transcribes English speech from a file called audio.mp3:

import os

from openai import OpenAI

client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)

with open("audio.mp3", "rb") as audio_file:
   transcription = client.audio.transcriptions.create(
       model="openai/gpt-4o-mini-transcribe",
       file=audio_file,
       language="en",
       response_format="json",
   )

print(transcription.text)

Parameters:

model: the transcription model to use
file: the audio file to convert to text
language: the language of the audio. If not specified, the model detects it automatically.
response_format: the response format—json or text

Passing Context

You can use the prompt parameter to give the model relevant terms, extra context, and requirements for the output. You can also ask it to remove filler words:

import os
from openai import OpenAI

client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)

with open("meeting.mp3", "rb") as audio_file:
   transcription = client.audio.transcriptions.create(
       model="openai/gpt-4o-transcribe",
       file=audio_file,
       language="en",
       response_format="json",
       prompt=(
           "The recording is about Kubernetes and cloud infrastructure. "
           "The conversation mentions Hostman, kubectl, and Ingress. "
           "Remove filler words and interjections from the transcript. "
           "Keep the meaning intact and don't paraphrase the rest of the speech."
       ),
   )

print(transcription.text)

In this example, prompt helps the model recognize technical terms and sets requirements for the final text. This parameter doesn't replace the audio content—it adds context and instructions on top of it.

Transcribing Audio From a Microphone

AI Gateway doesn't support real-time transcription. To get results close to real time, you can record audio in short chunks and send each chunk to the API separately.

Install the additional libraries:

pip install numpy sounddevice

The example below records audio in three-second chunks. A small overlap is added between adjacent chunks so the model doesn't lose words at the boundary between recordings:

import io
import os
import queue
import threading
import time
import wave
from dataclasses import dataclass

import numpy as np
import sounddevice as sd
from openai import OpenAI

MODEL = "openai/gpt-4o-mini-transcribe"

SAMPLE_RATE = 16_000
CHANNELS = 1
SAMPLE_WIDTH_BYTES = 2

CHUNK_SECONDS = 3.0
OVERLAP_SECONDS = 0.4
FRAMES_PER_BLOCK = 1_600

client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)

@dataclass
class AudioChunk:
   index: int
   pcm_data: bytes

audio_queue: queue.Queue[AudioChunk | None] = queue.Queue(maxsize=10)
stop_event = threading.Event()

def pcm_to_wav_bytes(pcm_data: bytes) -> io.BytesIO:
   wav_buffer = io.BytesIO()

   with wave.open(wav_buffer, "wb") as wav_file:
       wav_file.setnchannels(CHANNELS)
       wav_file.setsampwidth(SAMPLE_WIDTH_BYTES)
       wav_file.setframerate(SAMPLE_RATE)
       wav_file.writeframes(pcm_data)

   wav_buffer.seek(0)
   wav_buffer.name = "chunk.wav"

   return wav_buffer

def transcription_worker() -> None:
   while True:
       chunk = audio_queue.get()

       if chunk is None:
           audio_queue.task_done()
           break

       try:
           wav_file = pcm_to_wav_bytes(chunk.pcm_data)
           started_at = time.perf_counter()

           transcription = client.audio.transcriptions.create(
               model=MODEL,
               file=wav_file,
               language="en",
               response_format="json",
           )

           elapsed = time.perf_counter() - started_at
           text = getattr(transcription, "text", "").strip()

           if text:
               print(
                   f"\n[{chunk.index:04d}] "
                   f"({elapsed:.2f} sec.) {text}",
                   flush=True,
               )
           else:
               print(
                   f"\n[{chunk.index:04d}] "
                   f"({elapsed:.2f} sec.) [silence]",
                   flush=True,
               )

       except Exception as exc:
           print(
               f"\nError transcribing chunk "
               f"{chunk.index}: {exc}",
               flush=True,
           )

       finally:
           audio_queue.task_done()


def record_microphone() -> None:
   chunk_samples = int(SAMPLE_RATE * CHUNK_SECONDS)
   overlap_samples = int(SAMPLE_RATE * OVERLAP_SECONDS)

   accumulated = np.empty(0, dtype=np.int16)
   previous_tail = np.empty(0, dtype=np.int16)
   chunk_index = 1

   print("Speak now. Press Ctrl+C to stop.")

   with sd.InputStream(
       samplerate=SAMPLE_RATE,
       channels=CHANNELS,
       dtype="int16",
       blocksize=FRAMES_PER_BLOCK,
   ) as stream:
       while not stop_event.is_set():
           block, overflowed = stream.read(FRAMES_PER_BLOCK)

           if overflowed:
               print("\nWarning: audio buffer overflow.")

           accumulated = np.concatenate((accumulated, block[:, 0]))

           while len(accumulated) >= chunk_samples:
               current_samples = accumulated[:chunk_samples]
               accumulated = accumulated[chunk_samples:]

               if len(previous_tail) > 0:
                   samples_to_send = np.concatenate(
                       (previous_tail, current_samples)
                   )
               else:
                   samples_to_send = current_samples

               if overlap_samples > 0:
                   previous_tail = current_samples[-overlap_samples:].copy()
               else:
                   previous_tail = np.empty(0, dtype=np.int16)

               audio_chunk = AudioChunk(
                   index=chunk_index,
                   pcm_data=samples_to_send.astype(
                       "<i2",
                       copy=False,
                   ).tobytes(),
               )

               try:
                   audio_queue.put(audio_chunk, timeout=1)
                   print(".", end="", flush=True)
               except queue.Full:
                   print(
                       "\nQueue full: chunk skipped. "
                       "Increase CHUNK_SECONDS or check API speed."
                   )

               chunk_index += 1

       if len(accumulated) >= SAMPLE_RATE // 2:
           if len(previous_tail) > 0:
               accumulated = np.concatenate(
                   (previous_tail, accumulated)
               )

           audio_queue.put(
               AudioChunk(
                   index=chunk_index,
                   pcm_data=accumulated.astype(
                       "<i2",
                       copy=False,
                   ).tobytes(),
               )
           )


def main() -> None:
   worker = threading.Thread(
       target=transcription_worker,
       daemon=True,
   )
   worker.start()

   try:
       record_microphone()
   except KeyboardInterrupt:
       print("\nStopping recording...")
       stop_event.set()
   finally:
       audio_queue.put(None)
       audio_queue.join()
       worker.join(timeout=5)
       print("Done.")


if __name__ == "__main__":
   main()

You can adjust the CHUNK_SECONDS and OVERLAP_SECONDS values. Shorter chunks reduce latency but increase the number of requests. Because of the overlap, some words may repeat in the result—remove these duplicates during further text processing if needed.

Speech Synthesis Capabilities Copy link

Transcription Capabilities Copy link

How Tokens Are Counted Copy link

Limitations And Notes Copy link

Getting Started Copy link

Speech Synthesis Copy link

Generating an Audio File Copy link

Streaming Playback Copy link

Audio Transcription Copy link

Transcribing a File Copy link

Passing Context Copy link

Transcribing Audio From a Microphone Copy link