Most software costs finance already understands. A SaaS seat is a fixed monthly number. A reserved instance is a commitment with a known discount. Even a variable cloud bill moves within a range you can forecast from history. Large language model APIs break that intuition. They bill by the token (fractions of a word), and the number of tokens a single feature consumes can swing by an order of magnitude depending on how a prompt is written, how long a conversation runs, and which model a request happens to hit.
That is why LLM spend is often the line item nobody can explain. It does not appear as a predictable subscription; it accumulates request by request, buried inside a cloud or vendor bill, growing with usage in a way that has almost no relationship to headcount. The providers themselves describe the pricing plainly (Anthropic, OpenAI, and Google all publish per-token rates with input and output priced separately), but the plain description hides how quickly the meter can run. Finance teams that treat LLM APIs like another SaaS subscription consistently under-forecast them.
Input and output are not priced the same
The first thing that surprises finance is that a single API call has at least two prices inside it. Providers charge one rate for the tokens you send (the input: your prompt, plus any conversation history and retrieved documents) and a higher rate for the tokens the model generates (the output). Across Anthropic’s Claude, OpenAI’s GPT, and Google’s Gemini families, output tokens are consistently priced several times higher than input tokens, commonly in the range of three to five times.
That asymmetry changes how you reason about cost. A feature that reads a long document and returns a one-line answer is cheap. A feature that takes a short prompt and generates a long, detailed report is expensive, even though the user typed almost nothing. Two features with identical usage counts can differ in cost by 10x purely because of the input-to-output ratio. When you forecast, “number of requests” is not the unit that matters; tokens are, and output tokens dominate.
Newer “reasoning” models add a third category. Models that work through a problem step by step before answering generate internal reasoning tokens that the user never sees but the account still pays for, typically billed at the output rate. OpenAI documents these as reasoning tokens; Anthropic exposes them as extended thinking. A reasoning-heavy request can bill for far more output than the visible reply suggests, which is exactly the kind of invisible cost that erodes a budget built on visible behavior.
The hidden multiplier: context is re-billed every turn
The single most common reason LLM forecasts come in low is that teams forget the APIs are stateless. The model does not remember the previous message. To continue a conversation, the application re-sends the entire history (every prior question and answer) as input on the next call. A ten-turn conversation does not bill for ten messages; it bills for the first message once, the second message plus the first, the third plus the first two, and so on. Cost grows with the square of the conversation length, not linearly.
Retrieval-augmented generation compounds this. Every RAG call stuffs retrieved documents into the input, so a chatbot answering questions over a knowledge base may send thousands of context tokens with each user turn. A long system prompt, the standing instructions that define the assistant’s behavior, is re-sent on every single request too. Teams frequently discover that their largest token consumer is not user questions at all, but the fixed context riding along on every call.
Providers offer a direct lever here: prompt caching. When the same large context appears repeatedly, both Anthropic and OpenAI let you cache it so repeated reads are billed at a steep discount rather than full input price: Anthropic’s cached reads are a small fraction of the base input rate, and OpenAI discounts cached input automatically. For a workload with a heavy, stable system prompt or a shared document set, caching can cut input cost dramatically. Finance should ask a simple question of any LLM feature: is repeated context being cached, or are we paying full price to re-send the same tokens thousands of times a day?
Not every request needs the flagship model
The largest, most capable models are priced accordingly, and their per-token rates can be an order of magnitude above the smaller models in the same family. Yet a great deal of production traffic (classification, extraction, short summaries, routing decisions) does not need flagship reasoning. Sending it to the top model anyway is the LLM equivalent of running every batch job on your largest instance type.
The mature pattern is model routing: send simple requests to a small, cheap model and escalate only the genuinely hard ones to the flagship. Some teams add a cascade, where a cheap model attempts the task first and a more expensive model is called only when the first answer fails a confidence check. The savings are real precisely because the price gap between tiers is so wide. From a finance standpoint, the question is whether a single expensive model is handling all traffic by default, or whether the workload has been tiered so that spend follows difficulty.
Two more provider levers deserve a place in any cost review. Batch processing, where requests are submitted asynchronously and returned within a window rather than in real time, is offered by both OpenAI and Anthropic at roughly half the standard price, ideal for overnight or non-interactive workloads. And output length is directly controllable: because output is the expensive half, capping maximum response length and instructing models to be concise is one of the most immediate cost reductions available, with no change to the model or provider.
Cost per unit falls, but total spend keeps rising
There is a genuine tailwind here that finance should factor in. The cost of a given level of model capability has fallen sharply and continuously. Stanford’s AI Index has tracked the price of querying a model at a fixed performance level dropping by more than one hundredfold over roughly a year and a half, as providers release cheaper models and cut prices. On a per-token, per-capability basis, LLM inference is getting cheaper fast.
This is exactly why total spend still climbs. When something becomes an order of magnitude cheaper, organizations use far more of it, a dynamic economists have recognized for over a century. Cheaper tokens invite more features, longer contexts, more agents, and more users, and the aggregate bill grows even as each token costs less. Forecasting LLM spend off falling unit prices alone will always miss; the volume response swamps the price decline. The FinOps Foundation’s State of FinOps research has reflected this shift, with practitioners increasingly naming AI and machine-learning spend as a priority to bring under management alongside traditional cloud cost, and with forecasting cited year after year as the discipline’s hardest problem.
What finance should actually put in place
LLM API cost is controllable, but only with the same rigor finance already applies to cloud compute. A few practices separate the teams that keep it in bounds from the teams that get surprised.
Attribute spend to features and teams. Token usage that lands in one undifferentiated vendor bill cannot be governed. Tagging requests by feature, team, and environment, the same allocation discipline that makes a cloud bill legible, is what lets you see which AI feature is driving cost and whether that cost is justified by its value.
Forecast on tokens, not requests. Build the forecast from expected token volume, split by input and output, with explicit assumptions about conversation length, context size, and model mix. A forecast that says “we expect N requests” without those factors is not a forecast; it is a guess.
Review the levers on a schedule. Caching, batching, model routing, output caps, and reasoning-token usage are not set-and-forget. Model prices change, usage patterns shift, and new cheaper models appear regularly. A quarterly review of whether each lever is still being pulled is the difference between a bill that tracks value and one that drifts.
The organizations that stay ahead of LLM cost are not the ones that use less AI. They are the ones that made the token meter visible (attributed, forecasted, and governed) before it became the fastest-growing number on the bill.
CostDefender brings AI and LLM API spend into the same read-only cost intelligence finance already uses for cloud: attributing token cost to features and teams, surfacing the anomalies, and verifying the savings from every optimization. See how it works →