--- title: "Process Requests in Batches with Batch API | Hostman Docs" description: "Process large volumes of requests asynchronously with Batch API in AI Gateway. Upload a JSONL file, track the batch, and download results at half the cost." --- > For the complete documentation index for AI agents, see [llms.txt](https://hostman.com/llms.txt). Batch API lets you send multiple requests to a model in a single file and get the results once processing is complete. Requests run in the background and finish within 24 hours, usually sooner. Batch processing works well for tasks that don't need an immediate response, such as: - Translating large volumes of text - Classifying reviews and support tickets - Generating product descriptions - Creating text embeddings Batch API requests cost half as much as standard requests. To process requests in batches: 1. Prepare and upload a JSONL file with your requests. 2. Create a batch from the file. 3. Check the batch status until processing finishes. 4. Download the result and error files. Batch API is compatible with the OpenAI SDK. ## Supported Models and Limits Batch API is available only in AI Gateway. AI agent keys can't be used for batch processing. Only OpenAI models with batch processing enabled are supported: - Chat models, such as `openai/gpt-4.1-mini`: through the `/v1/chat/completions` endpoint - Embedding models from the `openai/text-embedding-*` family: through the `/v1/embeddings` endpoint JSONL file limits: | **Parameter** | **Limit** | | --- | --- | | Format | JSONL: one JSON object per line | | Number of requests | Up to 50,000 per file | | File size | Up to 50 MB | | Model | One per file | | Endpoint | One per file | | Processing time | Up to 24 hours | The following aren't supported inside requests: - The `stream` and `store` parameters - Tools other than `function`, such as `web_search` and `file_search` - Audio and image inputs - File references via `file_id` Requests with unsupported parameters fail with an error. ## Billing Batch API usage is billed by input and output tokens at 50% of the standard price for the selected model. ### Reserved Funds When you create a batch, the platform reserves funds on your account balance to cover the entire file. The reserved amount is the cost of input tokens plus the estimated cost of output tokens at the batch price. The number of output tokens is estimated separately for each line: - If `max_tokens` is set, its value is used. - If `max_tokens` is not set, the estimate is based on the model's maximum response length, `max_completion_tokens`. This maximum is applied to every request in the file, because the actual response length can't be known in advance. Without `max_tokens`, the reserved amount can be much higher than the final cost. To keep the reservation low, set `max_tokens` based on the expected response length. The reserved amount is immediately deducted from your displayed balance and can't be used for other paid actions. The final cost of the batch is determined only after processing is complete. ### Funds for Other Services To create a batch, your account must have enough funds not only for the reservation but also for 30 hours of running your other services. This ensures that reserving funds for a batch doesn't lead to your other services being blocked. If your available balance is too low, the API returns a `402 Insufficient funds` error. To fix this, top up your balance, reduce the number of requests in the file, or lower the maximum response length. ### Charges After Processing When a batch finishes, the platform calculates the cost of the tokens actually used. Only successful requests with a `200` status code are billed. Failed requests are free. The unused part of the reservation is immediately returned to your available balance. The actual cost appears in your expense history as model usage. If a batch ends with one of the following statuses: - `expired` or `cancelled`: requests that completed successfully are billed, and the rest of the reservation is returned to your available balance. - `failed`: nothing is charged, and the entire reservation is returned to your available balance. ### Daily Key Limits Batch API respects the [daily limits of your API key](https://hostman.com/docs/ai-agents/ai-gateway/connection/#setting-a-token-limit). When you create a batch, the platform checks whether the estimated usage fits within the remaining token limit. The actual usage is counted toward the limit after processing is complete. If a batch needs more tokens than the limit allows, its creation is rejected. A rejected batch doesn't consume any of the daily limit. ## Run Your First Batch This example translates two texts into Spanish with the `openai/gpt-4.1-mini` model. ### Prepare the Requests File 1. Create a file named `batch.jsonl`. 2. Add one request per line: ```shell {"custom_id":"1","method":"POST","url":"/v1/chat/completions","body":{"model":"openai/gpt-4.1-mini","messages":[{"role":"user","content":"Translate to Spanish: Hello!"}],"max_tokens":500}} {"custom_id":"2","method":"POST","url":"/v1/chat/completions","body":{"model":"openai/gpt-4.1-mini","messages":[{"role":"user","content":"Translate to Spanish: Thank you for your order."}],"max_tokens":500}} ``` 3. Save the file. Each JSON object must be on a single line. Don't wrap the requests in an array or add commas between lines. Parameters: - `custom_id`: A request ID that is unique within the file. Use it to match each response to its original request. - `method`: The HTTP method. - `url`: The endpoint that processes the request: `/v1/chat/completions` for chat models or `/v1/embeddings` for embeddings. The value must be the same on every line of the file. - `body`: The request body sent to the model. In this example, it contains the model name (`model`), the array of messages (`messages`), and the response length limit (`max_tokens`). For chat requests, we recommend setting `max_tokens` based on the expected response length. This parameter affects [the amount reserved](https://hostman.com/docs/ai-agents/ai-gateway/batch-api/#reserved-funds) when you create the batch. Without it, the reservation is higher. ### Upload the File 1. Send the file through the Files API with `purpose=batch`: ```shell curl --request POST \ --url https://ai-api.hostman.com/v1/files \ --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \ --form "purpose=batch" \ --form "file=@batch.jsonl" ``` Parameters: - - `purpose`: The purpose of the file. Set it to `batch` for batch processing. - `file`: The path to the file on your local machine, prefixed with `@`. 2. Copy the `id` value from the response. You need it to create the batch. For files with `purpose=batch`, the model is detected from the file contents. You don't need to pass the `target_model_names` parameter. ### Create a Batch 1. Send a `POST` request to the `/v1/batches` endpoint: ```shell curl --request POST \ --url https://ai-api.hostman.com/v1/batches \ --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "input_file_id": "FILE_ID", "endpoint": "/v1/chat/completions", "completion_window": "24h" }' ``` Parameters: - - `input_file_id`: The ID of the uploaded file. Replace `FILE_ID` with the `id` value from the previous step. - `endpoint`: The processing endpoint. It must match the `url` value in the file's lines. - `completion_window`: The processing time window. Set it to `24h`. 2. Copy the batch `id` from the response. The batch object is returned with the `validating` status. You need the ID to check the status and cancel the batch. Creating a batch [reserves funds](https://hostman.com/docs/ai-agents/ai-gateway/batch-api/#reserved-funds) on your balance to process the entire file. If your available balance is too low, the API returns a `402 Insufficient funds` error. ### Check the Batch Status 1. Send a `GET` request, providing your `BATCH_ID`: ```shell curl --request GET \ --url "https://ai-api.hostman.com/v1/batches/BATCH_ID" \ --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" ``` 2. Check the `status` field in the response. Possible values: | **Status** | **Description** | | --- | --- | | `validating` | The input data is being validated | | `in_progress` | The requests are being processed | | `finalizing` | The results are being prepared | | `completed` | Processing is complete. Individual requests may have errors | | `failed` | The batch failed | | `expired` | The processing time limit has passed | | `cancelled` | The batch was cancelled | 3. Repeat the request until processing finishes. A successful batch goes through `validating`, `in_progress`, `finalizing`, and `completed`. The `failed`, `expired`, and `cancelled` statuses also mean the batch is finished, so you don’t need to wait for `completed` in these cases. ### List Batches To see all batches available to the current key, send a `GET` request to `/v1/batches`: ```shell curl --request GET \ --url https://ai-api.hostman.com/v1/batches \ --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" ``` ### Download the Results The result file is stored for 30 days. Download it before it expires. 1. Check the batch status again to get its latest data. See [Check the Batch Status](https://hostman.com/docs/ai-agents/ai-gateway/batch-api/#check-the-batch-status). 2. In the response, find these fields: - `output_file_id`: The ID of the file with results. - `error_file_id`: The ID of the file with errors of individual requests. 3. If `output_file_id` is present, replace `OUTPUT_FILE_ID` with its value and download the results: ```shell curl --request GET \ --url "https://ai-api.hostman.com/v1/files/OUTPUT_FILE_ID/content" \ --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \ --output results.jsonl ``` The results are saved to `results.jsonl` in the current directory. Each line is a JSON object with these fields: - `custom_id`: The ID of the original request. - `response.status_code`: The HTTP status code of the response. - `response.body`: The body of the model response. - `error`: Error details, if any. The order of lines in the results may differ from the order in the input file. Match requests to responses by `custom_id`. 4. (Optional) If you have `jq` installed, print the ID and response text for each line: ```shell jq -r '[.custom_id, .response.body.choices[0].message.content] | @tsv' results.jsonl ``` For the two requests in this example, the output looks like this: ```shell 1 Hola! 2 Gracias por su pedido. ``` 5. If `error_file_id` is present, replace `ERROR_FILE_ID` with its value and download the error file: ```shell curl --request GET \ --url "https://ai-api.hostman.com/v1/files/ERROR_FILE_ID/content" \ --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" \ --output errors.jsonl ``` 6. Review `errors.jsonl`. It contains the error code and description for each failed request. Fix the requests and send them in a new batch. The `completed` status doesn't guarantee that every request succeeded, so always check the error file. If a batch is cancelled or expires, the requests that completed before that are available in the results. ## Create Embeddings in Batches To create embeddings in batches, prepare a separate JSONL file. Use a model with batch processing enabled and the `/v1/embeddings` endpoint. 1. Create a JSONL file with one request per line. For example, a request for the `openai/text-embedding-3-large` model looks like this: ```shell {"custom_id":"embedding-1","method":"POST","url":"/v1/embeddings","body":{"model":"openai/text-embedding-3-large","input":"Text to embed"}} ``` 2. Upload the file with `purpose=batch`. See [Upload the File](https://hostman.com/docs/ai-agents/ai-gateway/batch-api/#upload-the-file). 3. Create a batch and set `endpoint` to `/v1/embeddings`. See [Create a Batch](https://hostman.com/docs/ai-agents/ai-gateway/batch-api/#create-a-batch). 4. Check the status and download the results the same way as for chat requests. ## Cancel a Batch Send a `POST` request, providing your `BATCH_ID`: ```shell curl --request POST \ --url "https://ai-api.hostman.com/v1/batches/BATCH_ID/cancel" \ --header "Authorization: Bearer $HOSTMAN_AI_TOKEN" ``` Requests completed before the cancellation appear in the results and are billed. The remaining requests aren't processed or billed. The unused reservation is returned to your available balance.