Inference cost
The compute cost of running a trained model to answer a single query, measured in dollars per million tokens or per query.
Inference cost is what a provider pays to run a trained AI model against one query. Separate from training cost (one-time cost of building the model), it covers GPU time, memory bandwidth, and infrastructure overhead per request.
For frontier models in 2026, inference typically runs $2-15 per million input tokens and $5-45 per million output tokens. A heavy user consuming ~5M tokens per month costs the provider roughly $30-150 in raw inference — before margin, infrastructure, or overhead.
Why this matters for pricing: - A $20/mo subscription with unlimited frontier-model use is often subsidizing heavy users at a loss. This is why "unlimited" plans usually have hidden caps, quiet model downgrades, or tier changes over time. - Products with transparent tier structures align price to actual cost — less exciting marketing, but sustainable. - Pricing "wars" across chat products often reflect providers absorbing losses to acquire market share.
Inference cost is dropping fast. The same quality costs 5-10x less to serve in 2026 than in 2023 — a combination of better architectures (MoE), better inference software (batching, speculative decoding), and cheaper GPU generations.
Practical takeaway: if a product's price feels too low, it's either subsidized (won't last), routing to a cheaper model, or limiting usage in ways you don't immediately notice.