---
title: "Speech Synthesis and Recognition (TTS & STT) in AI Gateway | Hostman Docs"
description: "Learn how to use AI Gateway's speech synthesis and transcription models: generate audio from text, stream playback, transcribe files, and process near real-time audio from a microphone."
---

> For the complete documentation index for AI agents, see [llms.txt](https://hostman.com/llms.txt).

AI Gateway offers models for speech synthesis and speech recognition. You can use them to convert text into spoken audio, add voice responses to your applications, transcribe calls, and turn other audio recordings into text.

For speech synthesis, or TTS (Text-to-Speech), AI Gateway provides the `openai/gpt-4o-mini-tts` model.

For speech recognition, or STT (Speech-to-Text), you can use `openai/gpt-4o-mini-transcribe` and `openai/gpt-4o-transcribe`.

## Speech Synthesis Capabilities

The `openai/gpt-4o-mini-tts` model generates audio from text you provide. It supports different languages and lets you control how the speech sounds using an instruction—for example, setting the pace, intonation, emotional tone, or overall style.

You can choose from the following voices:

-   `alloy`
-   `ash`
-   `ballad`
-   `coral`
-   `echo`
-   `fable`
-   `nova`
-   `onyx`
-   `sage`
-   `shimmer`
-   `verse`
-   `marin`
-   `cedar`

You can listen to samples of each voice on [OpenAI.fm](https://openai.fm/).

The following audio formats are supported:

-   `mp3`: the default format
-   `opus`: suited for streaming and voice communication
-   `aac`: a compressed format commonly used on mobile devices
-   `flac`: lossless compression
-   `wav`: uncompressed audio
-   `pcm`: raw audio data with no file header

The `wav` and `pcm` formats work well for use cases where low latency matters. AI Gateway also supports streaming, so playback can start before the entire audio file has finished generating.

## Transcription Capabilities

The `openai/gpt-4o-mini-transcribe` and `openai/gpt-4o-transcribe` models convert speech from an audio file into text. You can either let the model detect the language automatically or specify it explicitly in the request.

Transcription supports the following file formats: `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `wav`, and `webm`. You can get the result back as JSON or as plain text.

You can also pass a prompt parameter with additional context in your request. This helps the model correctly recognize:

-   first and last names
-   product and company names
-   abbreviations
-   professional or technical terms
-   words that are hard to identify from pronunciation alone

If a recording is split into several chunks, you can pass the text of the previous chunk in the prompt parameter. This helps the model keep context between requests.

## How Tokens Are Counted

Audio models count both text tokens and audio tokens. Text tokens depend on the amount of text involved, while audio tokens depend on the length of the audio.

For speech synthesis (TTS models):

-   `input_tokens`: text tokens from the input text
-   `output_tokens`: audio tokens from the generated recording. One second of audio counts as one audio token.

For example, if the model generates four seconds of audio, that's four output audio tokens. The number of input text tokens depends on the length of the source text.

For transcription (STT models):

-   `input_tokens`: audio tokens from the source recording, roughly 10 tokens per second
-   `output_tokens`: text tokens from the transcript

For example, a 10-second recording uses roughly 100 input audio tokens. The number of output text tokens depends on the length of the transcript.

## Limitations And Notes

-   Audio models are only available through AI Gateway.
-   Streaming generation and playback are only supported for TTS.
-   Real-time transcription from a microphone isn't supported.
-   Transcription responses are available in JSON or plain text format.

## Getting Started

To use the audio models, you'll need to [create an AI Gateway API key](https://hostman.com/docs/ai-agents/ai-gateway/connection/#creating-an-api-key). 

The examples below use an environment variable called `HOSTMAN_AI_TOKEN` so you don't have to paste your key directly into your code. On Linux and macOS, you can set it with:

```shell
export HOSTMAN_AI_TOKEN="your_API_key"
```

Install the OpenAI library:

```shell
pip install openai
```

Create a client and point it to the AI Gateway base URL:

```py
import os
​
from openai import OpenAI
​
client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)
```

## Speech Synthesis

Speech synthesis uses the `audio.speech.create()` method.

### Generating an Audio File

In the example below, the model converts text to speech and saves the result to a file called `speech.mp3`:

```py
import os
from pathlib import Path
​
from openai import OpenAI
​
client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)
​
speech_file = Path("speech.mp3")
​
with client.audio.speech.with_streaming_response.create(
   model="openai/gpt-4o-mini-tts",
   voice="shimmer",
   input="Hello! This is a speech synthesis test.",
   instructions="Speak in a calm and friendly tone.",
   response_format="mp3",
) as response:
   response.stream_to_file(speech_file)
```

Parameters:

-   `model`: the speech synthesis model to use
-   `voice`: the voice used to read the text
-   `input`: the text to convert to speech
-   `instructions`: an instruction describing the speaking style
-   `response_format`: the format of the generated audio

### Streaming Playback

To start playback before generation finishes, use the `async` client together with `LocalAudioPlayer`.

To use `LocalAudioPlayer`, install the `numpy` and `sounddevice` libraries:

```shell
pip install numpy sounddevice
```

Example of streaming playback:

```py
import asyncio
import os
​
from openai import AsyncOpenAI
from openai.helpers import LocalAudioPlayer
​
client = AsyncOpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)
​
async def main() -> None:
   async with client.audio.speech.with_streaming_response.create(
       model="openai/gpt-4o-mini-tts",
       voice="shimmer",
       input="Hello! This is a streaming speech synthesis test.",
       response_format="pcm",
   ) as response:
       await LocalAudioPlayer().play(response)
​
if __name__ == "__main__":
   asyncio.run(main())
```

This example uses the `pcm` format, which lets you play the audio as it's received from the model.

## Audio Transcription

Transcription uses the `audio.transcriptions.create()` method.

### Transcribing a File

In the example below, the model transcribes English speech from a file called `audio.mp3`:

```py
import os
​
from openai import OpenAI
​
client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)
​
with open("audio.mp3", "rb") as audio_file:
   transcription = client.audio.transcriptions.create(
       model="openai/gpt-4o-mini-transcribe",
       file=audio_file,
       language="en",
       response_format="json",
   )
​
print(transcription.text)
```

Parameters:

-   `model`: the transcription model to use
-   `file`: the audio file to convert to text
-   `language`: the language of the audio. If not specified, the model detects it automatically.
-   `response_format`: the response format—json or text

### Passing Context

You can use the `prompt` parameter to give the model relevant terms, extra context, and requirements for the output. You can also ask it to remove filler words:

```py
import os
from openai import OpenAI
​
client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)
​
with open("meeting.mp3", "rb") as audio_file:
   transcription = client.audio.transcriptions.create(
       model="openai/gpt-4o-transcribe",
       file=audio_file,
       language="en",
       response_format="json",
       prompt=(
           "The recording is about Kubernetes and cloud infrastructure. "
           "The conversation mentions Hostman, kubectl, and Ingress. "
           "Remove filler words and interjections from the transcript. "
           "Keep the meaning intact and don't paraphrase the rest of the speech."
       ),
   )
​
print(transcription.text)
```

In this example, `prompt` helps the model recognize technical terms and sets requirements for the final text. This parameter doesn't replace the audio content—it adds context and instructions on top of it.

### Transcribing Audio From a Microphone

AI Gateway doesn't support real-time transcription. To get results close to real time, you can record audio in short chunks and send each chunk to the API separately.

Install the additional libraries:

```shell
pip install numpy sounddevice
```

The example below records audio in three-second chunks. A small overlap is added between adjacent chunks so the model doesn't lose words at the boundary between recordings:

```py
import io
import os
import queue
import threading
import time
import wave
from dataclasses import dataclass
​
import numpy as np
import sounddevice as sd
from openai import OpenAI
​
MODEL = "openai/gpt-4o-mini-transcribe"
​
SAMPLE_RATE = 16_000
CHANNELS = 1
SAMPLE_WIDTH_BYTES = 2
​
CHUNK_SECONDS = 3.0
OVERLAP_SECONDS = 0.4
FRAMES_PER_BLOCK = 1_600
​
client = OpenAI(
   api_key=os.environ["HOSTMAN_AI_TOKEN"],
   base_url="https://ai-api.hostman.com/v1",
)
​
@dataclass
class AudioChunk:
   index: int
   pcm_data: bytes
​
audio_queue: queue.Queue[AudioChunk | None] = queue.Queue(maxsize=10)
stop_event = threading.Event()
​
def pcm_to_wav_bytes(pcm_data: bytes) -> io.BytesIO:
   wav_buffer = io.BytesIO()
​
   with wave.open(wav_buffer, "wb") as wav_file:
       wav_file.setnchannels(CHANNELS)
       wav_file.setsampwidth(SAMPLE_WIDTH_BYTES)
       wav_file.setframerate(SAMPLE_RATE)
       wav_file.writeframes(pcm_data)
​
   wav_buffer.seek(0)
   wav_buffer.name = "chunk.wav"
​
   return wav_buffer
​
def transcription_worker() -> None:
   while True:
       chunk = audio_queue.get()
​
       if chunk is None:
           audio_queue.task_done()
           break
​
       try:
           wav_file = pcm_to_wav_bytes(chunk.pcm_data)
           started_at = time.perf_counter()
​
           transcription = client.audio.transcriptions.create(
               model=MODEL,
               file=wav_file,
               language="en",
               response_format="json",
           )
​
           elapsed = time.perf_counter() - started_at
           text = getattr(transcription, "text", "").strip()
​
           if text:
               print(
                   f"\n[{chunk.index:04d}] "
                   f"({elapsed:.2f} sec.) {text}",
                   flush=True,
               )
           else:
               print(
                   f"\n[{chunk.index:04d}] "
                   f"({elapsed:.2f} sec.) [silence]",
                   flush=True,
               )
​
       except Exception as exc:
           print(
               f"\nError transcribing chunk "
               f"{chunk.index}: {exc}",
               flush=True,
           )
​
       finally:
           audio_queue.task_done()
​
​
def record_microphone() -> None:
   chunk_samples = int(SAMPLE_RATE * CHUNK_SECONDS)
   overlap_samples = int(SAMPLE_RATE * OVERLAP_SECONDS)
​
   accumulated = np.empty(0, dtype=np.int16)
   previous_tail = np.empty(0, dtype=np.int16)
   chunk_index = 1
​
   print("Speak now. Press Ctrl+C to stop.")
​
   with sd.InputStream(
       samplerate=SAMPLE_RATE,
       channels=CHANNELS,
       dtype="int16",
       blocksize=FRAMES_PER_BLOCK,
   ) as stream:
       while not stop_event.is_set():
           block, overflowed = stream.read(FRAMES_PER_BLOCK)
​
           if overflowed:
               print("\nWarning: audio buffer overflow.")
​
           accumulated = np.concatenate((accumulated, block[:, 0]))
​
           while len(accumulated) >= chunk_samples:
               current_samples = accumulated[:chunk_samples]
               accumulated = accumulated[chunk_samples:]
​
               if len(previous_tail) > 0:
                   samples_to_send = np.concatenate(
                       (previous_tail, current_samples)
                   )
               else:
                   samples_to_send = current_samples
​
               if overlap_samples > 0:
                   previous_tail = current_samples[-overlap_samples:].copy()
               else:
                   previous_tail = np.empty(0, dtype=np.int16)
​
               audio_chunk = AudioChunk(
                   index=chunk_index,
                   pcm_data=samples_to_send.astype(
                       "<i2",
                       copy=False,
                   ).tobytes(),
               )
​
               try:
                   audio_queue.put(audio_chunk, timeout=1)
                   print(".", end="", flush=True)
               except queue.Full:
                   print(
                       "\nQueue full: chunk skipped. "
                       "Increase CHUNK_SECONDS or check API speed."
                   )
​
               chunk_index += 1
​
       if len(accumulated) >= SAMPLE_RATE // 2:
           if len(previous_tail) > 0:
               accumulated = np.concatenate(
                   (previous_tail, accumulated)
               )
​
           audio_queue.put(
               AudioChunk(
                   index=chunk_index,
                   pcm_data=accumulated.astype(
                       "<i2",
                       copy=False,
                   ).tobytes(),
               )
           )
​
​
def main() -> None:
   worker = threading.Thread(
       target=transcription_worker,
       daemon=True,
   )
   worker.start()
​
   try:
       record_microphone()
   except KeyboardInterrupt:
       print("\nStopping recording...")
       stop_event.set()
   finally:
       audio_queue.put(None)
       audio_queue.join()
       worker.join(timeout=5)
       print("Done.")
​
​
if __name__ == "__main__":
   main()
```

You can adjust the `CHUNK_SECONDS` and `OVERLAP_SECONDS` values. Shorter chunks reduce latency but increase the number of requests. Because of the overlap, some words may repeat in the result—remove these duplicates during further text processing if needed.