Token Caching


Caching reduces the cost of requests that repeat the same content at the start, such as a system prompt, documents, or conversation history. Tokens read from the cache are billed at a reduced rate.

It helps most when the same prefix is sent over and over at short intervals, for example:

  • Agent loops and coding assistants that frequently send requests with a large amount of repeated context
  • Long conversations where the full history is resent with every request

How Caching Works and What It Costs
Copy link

Most requests to a model start the same way: a system prompt, instructions, conversation history, attached documents. The provider caches that prefix and doesn't reprocess it on the next request. Caching doesn't change the model's output, only the speed and cost of processing.

The total number of input tokens stays the same. What changes is the price they're billed at. Input tokens fall into three categories:

Token type

Description

Price

Regular input

Tokens the model processes for the first time

Full input price

Cache read

The request prefix matches a recent request

Usually 5–20% of the input price

Cache write

Saving a prefix to the cache for the first time. Billed separately only for Claude and GPT-5.6+ models

125% of the input price

The cache is short-lived and usually expires a few minutes after the last request. If requests are infrequent, the cache may expire before the next one, and that request is processed from scratch. The exact retention time depends on the provider.

Cache read and write prices vary by model. You can find the exact values in the model catalog in the dashboard.

For Claude and GPT-6+ models, the first cache write costs more than regular input, but it happens once, while cache reads happen many times. The extra cost pays off from the second request with the same prefix. If you send a request only once, the total cost is slightly higher than without caching.

If a model has no separate cache price in the catalog, all input tokens are billed at the input price.

Enable Caching
Copy link

Automatic Caching
Copy link

For all models that support caching, except Claude, caching works automatically: the provider detects the repeated prefix on its own. You don't need to change your requests.

Caching with Claude
Copy link

Claude models cache a request only if it includes the cache_control parameter. Without it, all input tokens are billed at the full price.

Add cache_control to the last block of the static part of your request, such as the system prompt:

curl https://ai-api.hostman.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -d '{
    "model": "anthropic/claude-haiku-5-5",
    "messages": [
      {
        "role": "system",
        "content": [
          {
            "type": "text",
            "text": "<LONG_SYSTEM_PROMPT_OR_DOCUMENT>",
            "cache_control": {"type": "ephemeral"}
          }
        ]
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

The first request with cache_control writes the prefix to the cache. Later requests with the same prefix read it from the cache.

Check That Caching Works
Copy link

To confirm caching is working, send the same request twice a couple of minutes apart and compare the usage field in the responses.

In the first response, tokens are written to the cache:

"usage": {
  "prompt_tokens": 724,
  "prompt_tokens_details": {
    "cached_tokens": 0,
    "text_tokens": 24,
    "cache_write_tokens": 700
  },
  "cache_creation_input_tokens": 700,
  "cache_read_input_tokens": 0
}

In the second response, the same tokens are read from the cache:

"usage": {
  "prompt_tokens": 729,
  "prompt_tokens_details": {
    "cached_tokens": 700,
    "text_tokens": 29,
    "cache_write_tokens": 0
  },
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 700
}

The usage field comes from the model provider. Here’s how to read it:

  • prompt_tokens: Total input tokens, including cached ones
  • prompt_tokens_details.cached_tokens: Tokens read from the cache
  • prompt_tokens_details.cache_write_tokens: Tokens written to the cache
  • cache_read_input_tokens and cache_creation_input_tokens: The same two values for Claude models

If these values stay at zero after several requests with the same prefix, caching isn’t being used.

You can also track cache activity in the dashboard under AI Services → Activity.

Improve Your Cache Hit Rate
Copy link

Put static content (system prompt, instructions, documents) at the start of the request and dynamic content (the user's new question) at the end. Any change within the prefix invalidates the cache for it.

Limitations
Copy link

  • Web search results, images, and audio aren’t cached.
  • A prompt is cached only if its cacheable part meets a minimum length, which varies by model. For details, see the provider documentation: OpenAI | Anthropic.
  • Funds are reserved at the full price. Until a request is processed, the hold is calculated as if all input tokens were regular input. If a large share of the request is cached, the hold can be several times larger than the final charge, so your available balance may temporarily drop or even go negative. Amounts are recalculated within 1–3 hours, and the difference isn't charged.