Prompt caching

Prompt caching reuses cached prompt prefixes, reducing token consumption and response latency.

How it works#

When your requests contain a long, reused system prompt or context information:

• First request — all tokens are processed in full and the prompt prefix is cached
• Subsequent requests — on a cache hit, the cached tokens are no longer billed again
• Cache expiry — the cache has a TTL (typically 5-10 minutes); once expired it must be cached again

Cache support#

诺玛AI's model resources are provided by official cloud vendors such as AWS Bedrock, Azure OpenAI, Google Cloud, Alibaba Cloud, and Volcano Engine. Whenever a cloud vendor supports prompt caching for a model, 诺玛AI supports it too.

Cloud vendor

Representative models

Caching mechanism

AWS Bedrock

Claude series

Native prompt caching

Azure OpenAI

GPT-4o series

Automatic caching

Google Cloud

Gemini series

Context Caching

Alibaba Cloud

Qwen series

Platform-side caching

Volcano Engine

Doubao series

Platform-side caching

Cache support for a specific model follows each cloud vendor's official documentation. 诺玛AI passes through the cache-related parameters, with no extra configuration required.

Usage#

OpenAI protocol

Prompt caching for OpenAI models is automatic — it is enabled whenever a repeated prompt prefix is detected:

# The long system prompt is cached automatically
SYSTEM_PROMPT = """You are 诺玛AI's technical support assistant.
 
Here is the product information you need to know:
- 诺玛AI is an LLM gateway that supports many mainstream large models
- Supports the OpenAI / Anthropic / Gemini protocols
- ...
(more product knowledge omitted)
"""
 
# First request: cache the system prompt
response1 = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "Which models does 诺玛AI support?"}
    ]
)
 
# Second request: cache hit, faster and cheaper
response2 = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},  # Cache hit
        {"role": "user", "content": "How do I configure development tools?"}
    ]
)

Anthropic protocol

Anthropic models support explicit cache control:

Cost savings#

On a cache hit, the cached tokens are billed at a lower price; the savings vary by model:

• Anthropic Claude series — a cache hit can save about 90% of the input cost
• OpenAI GPT series — a cache hit can save about 50% of the input cost
• Google Gemini series — a cache hit can save about 50-75% of the input cost

The actual savings depend on the cache hit rate and each cloud vendor's billing policy. See the usage statistics in the 诺玛AI console for details.

Best practices#

• Put long text first — place unchanging parts such as the system prompt and knowledge base content at the start of messages
• Keep the prefix consistent — only an identical prefix will hit the cache
• Design the prompt structure well — separate the fixed parts from the changing parts

# Good design: fixed content first, variable content last
messages = [
    {"role": "system", "content": LONG_STATIC_PROMPT},   # Cacheable
    {"role": "user", "content": dynamic_question}          # Variable part
]
 
# Bad design: variable content interleaved with fixed content
messages = [
    {"role": "system", "content": f"Today is {date}. {LONG_PROMPT}"}  # Different every day, cannot be cached
]

Cache hits can be inspected in the `usage` field of the API response, and the cache hit rate is also shown in the usage statistics of the 诺玛AI console.

Last updated on June 23, 2026