Token Caching
Caching reduces the cost of requests that repeat the same content at the start, such as a system prompt, documents, or conversation history. Tokens read from the cache are billed at a reduced rate.
It helps most when the same prefix is sent over and over at short intervals, for example:
- Agent loops and coding assistants that frequently send requests with a large amount of repeated context
- Long conversations where the full history is resent with every request
How Caching Works and What It Costs Copy link
Most requests to a model start the same way: a system prompt, instructions, conversation history, attached documents. The provider caches that prefix and doesn't reprocess it on the next request. Caching doesn't change the model's output, only the speed and cost of processing.
The total number of input tokens stays the same. What changes is the price they're billed at. Input tokens fall into three categories:
|
Token type |
Description |
Price |
|
Regular input |
Tokens the model processes for the first time |
Full input price |
|
Cache read |
The request prefix matches a recent request |
Usually 5–20% of the input price |
|
Cache write |
Saving a prefix to the cache for the first time. Billed separately only for Claude and GPT-5.6+ models |
125% of the input price |
The cache is short-lived and usually expires a few minutes after the last request. If requests are infrequent, the cache may expire before the next one, and that request is processed from scratch. The exact retention time depends on the provider.
Cache read and write prices vary by model. You can find the exact values in the model catalog in the dashboard.
For Claude and GPT-6+ models, the first cache write costs more than regular input, but it happens once, while cache reads happen many times. The extra cost pays off from the second request with the same prefix. If you send a request only once, the total cost is slightly higher than without caching.
If a model has no separate cache price in the catalog, all input tokens are billed at the input price.
Enable Caching Copy link
Automatic Caching Copy link
For all models that support caching, except Claude, caching works automatically: the provider detects the repeated prefix on its own. You don't need to change your requests.
Caching with Claude Copy link
Claude models cache a request only if it includes the cache_control parameter. Without it, all input tokens are billed at the full price.
Add cache_control to the last block of the static part of your request, such as the system prompt:
curl https://ai-api.hostman.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR_API_KEY>" \
-d '{
"model": "anthropic/claude-haiku-5-5",
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "<LONG_SYSTEM_PROMPT_OR_DOCUMENT>",
"cache_control": {"type": "ephemeral"}
}
]
},
{
"role": "user",
"content": "Hello!"
}
]
}'
The first request with cache_control writes the prefix to the cache. Later requests with the same prefix read it from the cache.
Check That Caching Works Copy link
To confirm caching is working, send the same request twice a couple of minutes apart and compare the usage field in the responses.
In the first response, tokens are written to the cache:
"usage": {
"prompt_tokens": 724,
"prompt_tokens_details": {
"cached_tokens": 0,
"text_tokens": 24,
"cache_write_tokens": 700
},
"cache_creation_input_tokens": 700,
"cache_read_input_tokens": 0
}
In the second response, the same tokens are read from the cache:
"usage": {
"prompt_tokens": 729,
"prompt_tokens_details": {
"cached_tokens": 700,
"text_tokens": 29,
"cache_write_tokens": 0
},
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 700
}
The usage field comes from the model provider. Here’s how to read it:
prompt_tokens: Total input tokens, including cached onesprompt_tokens_details.cached_tokens: Tokens read from the cacheprompt_tokens_details.cache_write_tokens: Tokens written to the cachecache_read_input_tokensandcache_creation_input_tokens: The same two values for Claude models
If these values stay at zero after several requests with the same prefix, caching isn’t being used.
You can also track cache activity in the dashboard under AI Services → Activity.
Improve Your Cache Hit Rate Copy link
Put static content (system prompt, instructions, documents) at the start of the request and dynamic content (the user's new question) at the end. Any change within the prefix invalidates the cache for it.
Limitations Copy link
- Web search results, images, and audio aren’t cached.
- A prompt is cached only if its cacheable part meets a minimum length, which varies by model. For details, see the provider documentation: OpenAI | Anthropic.
- Funds are reserved at the full price. Until a request is processed, the hold is calculated as if all input tokens were regular input. If a large share of the request is cached, the hold can be several times larger than the final charge, so your available balance may temporarily drop or even go negative. Amounts are recalculated within 1–3 hours, and the difference isn't charged.