Process Requests in Batches

Batch API lets you send multiple requests to a model in a single file and get the results once processing is complete. Requests run in the background and finish within 24 hours, usually sooner.

Batch processing works well for tasks that don't need an immediate response, such as:

• Translating large volumes of text
• Classifying reviews and support tickets
• Generating product descriptions
• Creating text embeddings

Batch API requests cost half as much as standard requests.

To process requests in batches:

1. Prepare and upload a JSONL file with your requests.
2. Create a batch from the file.
3. Check the batch status until processing finishes.
4. Download the result and error files.

Batch API is compatible with the OpenAI SDK.

Supported Models and Limits

Copy link

Batch API is available only in AI Gateway. AI agent keys can't be used for batch processing.

Only OpenAI models with batch processing enabled are supported:

• Chat models, such as openai/gpt-4.1-mini: through the /v1/chat/completions endpoint
• Embedding models from the openai/text-embedding-* family: through the /v1/embeddings endpoint

JSONL file limits:

The following aren't supported inside requests:

• The stream and store parameters
• Tools other than function, such as web_search and file_search
• Audio and image inputs
• File references via file_id

Requests with unsupported parameters fail with an error.

Billing

Copy link

Batch API usage is billed by input and output tokens at 50% of the standard price for the selected model.

Reserved Funds

Copy link

When you create a batch, the platform reserves funds on your account balance to cover the entire file. The reserved amount is the cost of input tokens plus the estimated cost of output tokens at the batch price.

The number of output tokens is estimated separately for each line:

• If max_tokens is set, its value is used.
• If max_tokens is not set, the estimate is based on the model's maximum response length, max_completion_tokens. This maximum is applied to every request in the file, because the actual response length can't be known in advance.

Without max_tokens, the reserved amount can be much higher than the final cost. To keep the reservation low, set max_tokens based on the expected response length.

The reserved amount is immediately deducted from your displayed balance and can't be used for other paid actions. The final cost of the batch is determined only after processing is complete.

Funds for Other Services

Copy link

To create a batch, your account must have enough funds not only for the reservation but also for 30 hours of running your other services. This ensures that reserving funds for a batch doesn't lead to your other services being blocked.

If your available balance is too low, the API returns a 402 Insufficient funds error. To fix this, top up your balance, reduce the number of requests in the file, or lower the maximum response length.

Charges After Processing

Copy link

When a batch finishes, the platform calculates the cost of the tokens actually used. Only successful requests with a 200 status code are billed. Failed requests are free.

The unused part of the reservation is immediately returned to your available balance. The actual cost appears in your expense history as model usage.

If a batch ends with one of the following statuses:

• expired or cancelled: requests that completed successfully are billed, and the rest of the reservation is returned to your available balance.
• failed: nothing is charged, and the entire reservation is returned to your available balance.

Daily Key Limits

Copy link

Batch API respects the daily limits of your API key. When you create a batch, the platform checks whether the estimated usage fits within the remaining token limit. The actual usage is counted toward the limit after processing is complete.

If a batch needs more tokens than the limit allows, its creation is rejected. A rejected batch doesn't consume any of the daily limit.

Run Your First Batch

Copy link

This example translates two texts into Spanish with the openai/gpt-4.1-mini model.

Prepare the Requests File

Copy link

1. Create a file named batch.jsonl.
2. Add one request per line:

{"custom_id":"1","method":"POST","url":"/v1/chat/completions","body":{"model":"openai/gpt-4.1-mini","messages":[{"role":"user","content":"Translate to Spanish: Hello!"}],"max_tokens":500}}
{"custom_id":"2","method":"POST","url":"/v1/chat/completions","body":{"model":"openai/gpt-4.1-mini","messages":[{"role":"user","content":"Translate to Spanish: Thank you for your order."}],"max_tokens":500}}

3. Save the file.

Each JSON object must be on a single line. Don't wrap the requests in an array or add commas between lines.

Parameters:

• custom_id: A request ID that is unique within the file. Use it to match each response to its original request.
• method: The HTTP method.
• url: The endpoint that processes the request: /v1/chat/completions for chat models or /v1/embeddings for embeddings. The value must be the same on every line of the file.
• body: The request body sent to the model. In this example, it contains the model name (model), the array of messages (messages), and the response length limit (max_tokens).

For chat requests, we recommend setting max_tokens based on the expected response length. This parameter affects the amount reserved when you create the batch. Without it, the reservation is higher.

Upload the File

Copy link

1. Send the file through the Files API with purpose=batch:

curl --request POST \
  --url https://ai-api.hostman.com/v1/files \
  --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \
  --form "purpose=batch" \
  --form "file=@batch.jsonl"

 Parameters:

• • purpose: The purpose of the file. Set it to batch for batch processing.
  • file: The path to the file on your local machine, prefixed with @.

2. Copy the id value from the response. You need it to create the batch.

For files with purpose=batch, the model is detected from the file contents. You don't need to pass the target_model_names parameter.

Create a Batch

Copy link

1. Send a POST request to the /v1/batches endpoint:

curl --request POST \
  --url https://ai-api.hostman.com/v1/batches \
  --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "input_file_id": "FILE_ID",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

 Parameters:

• • input_file_id: The ID of the uploaded file. Replace FILE_ID with the id value from the previous step.
  • endpoint: The processing endpoint. It must match the url value in the file's lines.
  • completion_window: The processing time window. Set it to 24h.

2. Copy the batch id from the response. The batch object is returned with the validating status. You need the ID to check the status and cancel the batch.

Creating a batch reserves funds on your balance to process the entire file. If your available balance is too low, the API returns a 402 Insufficient funds error.

Check the Batch Status

Copy link

1. Send a GET request, providing your BATCH_ID:

curl --request GET \
  --url "https://ai-api.hostman.com/v1/batches/BATCH_ID" \
  --header "Authorization: Bearer $HOSTMAN_AI_TOKEN"

2. Check the status field in the response. Possible values:

3. Repeat the request until processing finishes.

A successful batch goes through validating, in_progress, finalizing, and completed. The failed, expired, and cancelled statuses also mean the batch is finished, so you don’t need to wait for completed in these cases.

List Batches

Copy link

To see all batches available to the current key, send a GET request to /v1/batches:

curl --request GET \
  --url https://ai-api.hostman.com/v1/batches \
  --header "Authorization: Bearer $HOSTMAN_AI_TOKEN"

Download the Results

Copy link

The result file is stored for 30 days. Download it before it expires.

1. Check the batch status again to get its latest data. See Check the Batch Status.
2. In the response, find these fields:
   
   • output_file_id: The ID of the file with results.
   • error_file_id: The ID of the file with errors of individual requests.
3. If output_file_id is present, replace OUTPUT_FILE_ID with its value and download the results:

curl --request GET \
  --url "https://ai-api.hostman.com/v1/files/OUTPUT_FILE_ID/content" \
  --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \
  --output results.jsonl

The results are saved to results.jsonl in the current directory. Each line is a JSON object with these fields:

• custom_id: The ID of the original request.
• response.status_code: The HTTP status code of the response.
• response.body: The body of the model response.
• error: Error details, if any.

The order of lines in the results may differ from the order in the input file. Match requests to responses by custom_id.

4. (Optional) If you have jq installed, print the ID and response text for each line:

jq -r '[.custom_id, .response.body.choices[0].message.content] | @tsv' results.jsonl

For the two requests in this example, the output looks like this:

1  Hola!
2  Gracias por su pedido.

5. If error_file_id is present, replace ERROR_FILE_ID with its value and download the error file:

curl --request GET \
  --url "https://ai-api.hostman.com/v1/files/ERROR_FILE_ID/content" \
  --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \
  --output errors.jsonl

6. Review errors.jsonl. It contains the error code and description for each failed request. Fix the requests and send them in a new batch.

The completed status doesn't guarantee that every request succeeded, so always check the error file.

If a batch is cancelled or expires, the requests that completed before that are available in the results.

Create Embeddings in Batches

Copy link

To create embeddings in batches, prepare a separate JSONL file. Use a model with batch processing enabled and the /v1/embeddings endpoint.

1. Create a JSONL file with one request per line. For example, a request for the openai/text-embedding-3-large model looks like this:

{"custom_id":"embedding-1","method":"POST","url":"/v1/embeddings","body":{"model":"openai/text-embedding-3-large","input":"Text to embed"}}

2. Upload the file with purpose=batch. See Upload the File.
3. Create a batch and set endpoint to /v1/embeddings. See Create a Batch.
4. Check the status and download the results the same way as for chat requests.

Cancel a Batch

Copy link

Send a POST request, providing your BATCH_ID:

curl --request POST \
  --url "https://ai-api.hostman.com/v1/batches/BATCH_ID/cancel" \
  --header "Authorization: Bearer $HOSTMAN_AI_TOKEN"

Requests completed before the cancellation appear in the results and are billed. The remaining requests aren't processed or billed. The unused reservation is returned to your available balance.