LLM Token Budgeting: Sizing Prompts, Context, and Cost

Every LLM call is billed by the token, and every model has a hard context limit measured in tokens. If you don't budget them, two things happen: your costs drift unpredictably, and requests start failing when a stuffed prompt overflows the window. Token budgeting is the discipline of knowing — before you ship — roughly how many tokens each request uses and what that costs at scale. Here's how to do it.

Tokens 101 (just enough)

A token is a chunk of text — for English, roughly 4 characters or ¾ of a word on average. Both your input (system prompt + context + user message) and the model's output are counted, and providers usually price them separately (output often costs more than input).

Two numbers govern every request:

• Cost = input_tokens × input_price + output_tokens × output_price
• Fit = input_tokens + max_output_tokens must stay under the model's context window

Where your tokens actually go

For a typical RAG or agent request, the input is dominated by things you control:

• System prompt — sent on every call. Trim it; it adds up fast at scale.
• Retrieved context (RAG) — usually the biggest chunk. topK × chunk size = your context bill.
• Conversation history — grows every turn if you don't window it.
• Tool definitions — each tool's schema costs tokens on every call.
• The user message — usually the smallest part.

Output tokens are governed by the task and your max_tokens cap.

The levers that control the budget

1. Cap output — set max_tokens to what the task actually needs. Uncapped output is the #1 surprise cost.
2. Right-size RAG — a topK of 4–6 focused chunks beats 20 mediocre ones, and costs a quarter as much. Tune chunking so each chunk earns its tokens.
3. Window the history — keep the last N turns (or summarize older ones), don't replay the whole conversation.
4. Trim the system prompt — every wasted sentence is billed on every request. Move rarely-needed instructions into tools or conditionals.
5. Use prompt caching — for stable prefixes (system prompt, big context), prompt caching can cut input cost dramatically on repeated calls.
6. Match the tier to the task — high-volume extraction doesn't need a frontier model; a fast/cheap tier slashes the per-token price.

A simple budgeting method

1. Estimate one request. Add up system prompt + typical context + history + expected output tokens (use the token counter for the text-heavy parts).
2. Cost it. Multiply by the model tier's input/output prices — the LLM Cost Estimator does this across models.
3. Scale it. Multiply by expected requests/day × 30 for a monthly figure.
4. Add headroom + set caps. Add ~20% buffer, set max_tokens, and alert on anomalies.
5. Meter in production. Extract the usage token counts from each response and track per feature/consumer — an AI gateway makes this centralized.

FAQ

How accurate is the ~4-chars-per-token rule? Good enough for planning English prose. Code, non-English text, and rare words tokenize differently — use it for budgets, not exact billing.

Input or output — which should I optimize first? Depends on your ratio. RAG/agent apps are usually input-heavy (context dominates) — trim context and cache. Chat/generation apps are output-heavy — cap max_tokens.

Does a bigger context window mean I should use it? No. Just because a model accepts 1M tokens doesn't mean you should send them — you pay for every one, and more context often dilutes the answer. Send the least context that answers the question.

How do I stop one team blowing the budget? Meter and quota per consumer at an AI gateway, so each team has an enforceable token budget.

Token budgeting turns "why is the bill unpredictable?" into a number you set on purpose. Estimate a request, cost it, scale it, cap it, and meter it — and your LLM spend becomes a lever you control instead of a surprise.

Try it now: estimate your prompt with the LLM Token Counter, then price it with the LLM Cost Estimator. More: prompt caching and LLM observability.

Need help getting an AI feature's unit economics under control? Let's talk.