The honest answer is: it depends enormously on what you are building and how you are building it, and most cost estimates you find online are either too optimistic or based on use cases that do not match yours. Here is a more grounded way to think through the numbers.
The main cost buckets
Inference costs are what most people mean when they ask this question. This is what it costs to actually run the model on new inputs in production. For API-based models, this is per-token or per-request pricing that shows up on a monthly bill. For self-hosted models, it is your compute costs, primarily GPU time, plus the engineering overhead of running the infrastructure reliably.
Training costs are separate and often substantially larger. Training a genuinely large model from scratch can cost anywhere from tens of thousands to millions of dollars in compute, depending on the model size and training duration. Fine-tuning an existing model is considerably cheaper, often in the range of hundreds to low thousands of dollars depending on dataset size and the model you are starting from.
What drives inference costs up
Model size is the dominant factor. Larger models cost more to run per request and respond more slowly. If your specific task does not actually require a large model, using a smaller one is almost always the right call: not just for cost reasons but for latency, which directly affects user experience in customer-facing applications.
Request volume and concurrency matter significantly. A model handling ten requests per second needs very different infrastructure from one handling a hundred simultaneous requests. When planning capacity, peak load matters more than average load. Building for average and getting caught out by spikes is an expensive mistake.
Hidden costs people miss
Beyond the model itself, there are costs that often get overlooked in early estimates. Data storage and retrieval for context. Embedding generation if you are building retrieval-augmented systems. Monitoring and logging infrastructure. The developer time spent on prompt engineering and iterating on output quality. These add up meaningfully at scale and should be part of any realistic budget.
A practical framework by stage
For early-stage projects: use a paid API, budget a few hundred dollars per month for experimentation, and do not optimize prematurely before you have real usage data to inform decisions. For mid-scale products: evaluate whether a smaller fine-tuned model can reach the same performance threshold as a larger general model for your specific task. For high-volume production: model cost optimization becomes a genuine engineering discipline worth dedicated attention, and the numbers can justify significant investment in that work.
What actual production systems cost at different scales
A realistic cost range for a production AI feature at modest scale, handling 50,000 requests per month using a mid-tier API model, sits in the range of USD 50 to 300 per month depending on prompt length and model choice. At ten times that volume, the monthly API bill typically reaches USD 500 to 3,000. At that point, evaluating whether a fine-tuned smaller model could handle the same task at lower cost per token becomes worth the engineering investment. At millions of monthly requests, infrastructure optimization becomes a genuine engineering discipline that typically pays for itself many times over.
The hidden cost category that consistently surprises teams scaling from prototype to production is retrieval and storage for context-augmented systems. Embedding generation, vector database storage and query costs, and the caching infrastructure needed to keep latency acceptable all add meaningful line items that do not appear in simple per-token pricing estimates. Building a realistic cost model before scaling to production prevents the common situation where a feature that looked economically viable at small scale becomes unsustainable at the volume needed to justify the development investment.
References & Further Reading