Configure Fallback Models

Fallback models let AI Gateway automatically reroute a request to another language model if the primary model returns an error or doesn't respond in time.

You can select up to two fallback models for each AI Gateway API key and set the order in which they're called.

The primary model is defined in each request by the model field. In the key settings, you select only the fallback models. The same chain applies to all requests made with that key.

Fallback models must be text models. Local models aren't supported, and you can't add the same model to the chain twice.

Add Fallback Models

Copy link

1. Go to AI Services → AI Gateway → API Keys and click the key you want to configure.
2. Open the Management tab.
3. In the Fallback Models row, click Select. If fallback models are already configured, click Change.
4. In the Model 2 field, select the first fallback model.
5. (Optional) In the Model 3 field, select the second fallback model. 
6. Click Save.

On the Management tab, you will now see the selected chain.

Edit Fallback Models

Copy link

1. Go to AI Services → AI Gateway → API Keys and click the key you want to configure.
2. Open the Management tab.
3. In the Fallback Models row, click Change.
4. Select different models in the Model 2 and Model 3 fields.
5. Click Save.

Remove a Fallback Model

Copy link

1. Go to AI Services → AI Gateway → API Keys and click the key you want to configure.
2. Open the Management tab.
3. In the Fallback Models row, click Change.
4. Click the cross in the field of the model you want to remove. If you remove Model 2, the model selected in Model 3 moves to Model 2.
5. Click Save.

Disable Fallback

Copy link

1. Go to AI Services → AI Gateway → API Keys and click the key you want to configure.
2. Open the Management tab.
3. In the Fallback Models row, click Change.
4. Clear both the Model 2 and Model 3 fields.
5. Click Save.

On the Management tab, the Fallback Models row now shows Not selected, and AI Gateway no longer switches models automatically.

How Fallback Works

Copy link

AI Gateway processes each request in the following order:

1. Sends the request to the primary model specified in the model field.
2. If the primary model doesn't respond successfully, sends the request to the first fallback model (Model 2).
3. If the first fallback model also returns an error, sends the request to Model 3, if selected.
4. If no model responds successfully, returns an error.

If a fallback model is the same as the primary model of the request, AI Gateway skips it and moves to the next one. Each fallback model is called at most once per request.

When Fallback Is Triggered

Copy link

AI Gateway switches to the next model when:

• The model provider is unavailable or returns a 5xx error.
• The request times out.
• The provider rate-limits the request (429 error).
• The request exceeds the model's context window.
• The provider rejects the request as invalid (400 error).

When Fallback Is Not Triggered

Copy link

AI Gateway doesn't switch to the next model when:

• The model returns a successful response, including a refusal. For example, "I can't discuss this topic" isn't treated as an error.
• The key's limit for the requested model is exhausted. AI Gateway checks the limit before calling the model, so the API returns a 422 error even if the fallback model's limit isn't exhausted. To continue, increase the limit for the requested model or specify a different model in the request.
• A streaming response has already started. Once the first chunk arrives, AI Gateway doesn't switch models.

Supported Requests

Copy link

Fallback works for requests to /v1/chat/completions and /v1/responses.

Fallback isn't supported for:

• Embeddings
• Image generation
• Speech generation and transcription
• Anthropic-format requests to /v1/messages
• The legacy /completions endpoint

Billing

Copy link

A request is billed at the rate of the model that generated the response. A failed attempt by the primary model isn't billed and doesn't count toward its limit.

Token usage counts toward the limit of the model that responded and toward the key's overall limit.

Identify the Model That Responded

Copy link

1. In the API response, check the model field. It contains the name of the model that actually processed the request.
2. Check the x-hostman-attempted-fallbacks HTTP header. A value greater than zero means AI Gateway switched to a fallback model. The value indicates that a switch occurred, not how many steps of the chain were used.

For example, if you send a request to gemini/gemini-3.1-flash-lite, the response may come from the fallback model Claude 4.5 Haiku. Here's a shortened example of the response:

HTTP/1.1 200 OK
x-hostman-attempted-fallbacks: 1

{
  "model": "claude-haiku-4-5-20251001",
  "choices": [
    {
      "message": {
        "content": "Yes"
      }
    }
  ]
}