The True Cost of Private LLM Inference: A Total-Cost-of-Ownership Model for On-Prem Deployments
Somewhere around the third "we should just run our own GPUs" conversation of the quarter, we started keeping a spreadsheet. Not because we disagree with the instinct — we run private inference for several clients and often recommend it — but because the comparisons people bring to the table are almost always wrong in the same direction. They compare GPU sticker prices to API token prices and conclude on-prem is either obviously cheap or obviously insane.
Both conclusions are wrong. The real answer is that on-prem inference has a cost structure with sharp thresholds: below certain volumes and utilization levels it is strictly more expensive than APIs, and above them it can be dramatically cheaper — provided you count everything.
This article is the model we use with clients. It is deliberately opinionated about numbers, because a TCO model full of hand-waving is useless. Where we state a figure, treat it as a planning anchor for 2026, not gospel — your colo contract, staffing situation, and workload mix will move things.
Why this decision matters more in 2026
The macro trend first: by most industry estimates, roughly 55% of enterprise inference workloads now run on-prem or at the edge, up from around 12% in 2023. Two forces drove that shift, and neither is nostalgia for data centers.
The first is unit economics at scale. Frontier API pricing fell, but enterprise token consumption grew faster than most budgets. Organizations that crossed a few billion tokens per month started doing the math and found that at their volumes, amortized GPU cost undercut API list prices by 2-5x — if they could keep the hardware busy.
The second is regulation and sovereignty. The EU AI Act came into force in stages through 2024-2026, and alongside GDPR it made data residency and documentation obligations a board-level concern. For legal, healthcare, defense, and financial workloads, "the data never leaves our perimeter" is not a nice-to-have; it is the procurement requirement. Several of our clients run on-prem purely for this reason and would do so even if it cost more.
So the question is no longer "is on-prem a niche?" It is "does the math work for your workload?" That is what the rest of this article answers.
The cost model: six line items, no hiding
We model annual TCO as:
TCO = Capex amortization + Power and cooling + Colocation/space + MLOps staffing + Model refresh + Downtime/overhead reserve
Let's take each in turn, then run a worked example.
1. Capex and amortization
A production-grade inference node in 2026 — say a server with eight high-end GPUs (H100-class or the current open-market equivalents), NVLink, 2TB RAM, and fast local storage — runs somewhere in the band of $250k-$400k depending on vendor, memory configuration, and how badly the market is supply-constrained this quarter.
The mistake is amortizing it like a rack server. GPU requirements grow with model scale: a node that comfortably serves today's 70B-class models at good throughput may struggle with the 2028 generation. We amortize inference GPU capex over three years, not five. Some finance teams push for four; that is defensible for accounting, but plan the hardware refresh at three.
That gives you, for a $320k node:
Annual capex amortization = 320,000 / 3 = ~$107k per year
2. Utilization: the number that dominates everything
Here is where most back-of-envelope calculations die. A GPU's advertised throughput — the tokens-per-second figure in the vendor deck — assumes a saturated batch. Production traffic is not saturated. It has diurnal peaks, idle nights, a second model replica sitting warm for failover, and bursty agentic workloads that spike context lengths unpredictably.
We plan around 30-50% real-world utilization for a single-tenant enterprise deployment. Well-run internal platforms with good queueing and continuous batching (vLLM and SGLang both do a lot of heavy lifting here) can reach 60-70% on steady, batchable workloads. Anything above that across a full month means you are dropping requests at peak.
The throughput math itself, for a 70B-class model in FP8 on a single 8-GPU node with continuous batching:
- Peak aggregate throughput: ~6,000-10,000 output tokens/sec (implementation- and stack-dependent)
- At 40% utilization: ~2,400-4,000 effective tokens/sec
- Over a 730-hour month: roughly 6-10 billion output tokens per month from one node
Hold that number. It becomes the denominator for every per-token cost you compute, and it is why utilization is the single highest-leverage operational metric you have.
3. Power and cooling
An 8-GPU H100-class node draws roughly 6-10 kW at full load; call it ~7 kW average including the host. At 40% utilization it does not draw 40% of peak power — GPUs idle hot — so budget closer to 5-6 kW sustained.
Annual energy = 5.5 kW × 8,760 h ≈ 48,000 kWh At $0.10/kWh: ~$4,800/year in electricity
Then apply a PUE (power usage effectiveness) multiplier for the facility. Good colos run 1.2-1.4; a closet in your office runs 2.0+. At PUE 1.3, total power cost is ~$6,200/year per node. Not the dominant line item, but never zero, and it doubles if your electricity is expensive.
4. Space, colocation, and the physical layer
Most organizations do not build data centers for this; they colo. Colo pricing for high-density racks runs roughly $1,500-$3,500 per kW per month all-in (space, power delivery, cooling, network). For our 7 kW node that is $10k-$25k/month if you rent by the kW with low commitment — which is why committed contracts and higher density matter. In a shared rack under an existing contract, the marginal cost of one node is far lower, often $3k-$6k/month. This line item has the widest variance of any in the model; get a real quote.
5. MLOps staffing: the line everyone forgets
Someone has to keep the serving stack alive: upgrade vLLM, babysit the quantization of the next model release, handle the node when an InfiniBand link flakes at 2am, run the eval harness before every model swap, own the autoscaling policy. This is a real fraction of a senior engineer's time.
Our planning figure for a single-node-to-small-cluster deployment: 0.3-0.5 FTE of an experienced MLOps/infra engineer. Fully loaded, that is $60k-$120k per year. On a one-node deployment, staffing is frequently the largest single cost line — often more than the hardware amortization. This is the honest reason small on-prem deployments often lose to APIs even when the electricity math works.
6. Model refresh cadence
Open-weight frontier models turn over every 6-12 months. Each generational jump means: re-benchmarking candidate models, re-tuning prompts and RAG retrieval parameters, re-validating guardrails, regression-testing against your eval set, and possibly re-quantizing for your memory budget. Budget $15k-$40k of engineering time per year for a mid-complexity production system, plus the risk that the next model you actually need does not fit your current VRAM. The hardware-refresh risk lives here too: assume models want 2-3x more memory within three years.
A worked example
Let's run the full model for a realistic mid-size deployment: two 8-GPU nodes ($640k total capex) serving a mix of RAG chat and document-processing agents for a 2,000-person company.
Assumptions: three-year amortization, 40% sustained utilization, ~15,000 output tokens/sec peak aggregate across both nodes → ~6,000 effective tokens/sec → roughly 15 billion output tokens/month. Colocation under a committed contract at $4k/month/node. 0.5 FTE MLOps.
| Cost line | Assumption | Annual cost | % of total |
|---|---|---|---|
| Capex amortization | $640k over 3 years | $213,000 | 47% |
| MLOps staffing | 0.5 FTE fully loaded | $90,000 | 20% |
| Colocation | 2 nodes × $4k/mo | $96,000 | 21% |
| Power and cooling | 12 kW avg, PUE 1.3, $0.10/kWh | $13,700 | 3% |
| Model refresh | Re-eval, tuning, testing | $25,000 | 5% |
| Downtime reserve | Spares, incidents, ~3% | $15,000 | 3% |
| Total annual TCO | ~$453,000 | 100% |
Effective cost = 453,000 / 180e9 tokens ≈ $2.5 per million output tokens
Compare against API pricing for comparable open-weight or frontier models: 2026 pricing for strong frontier-class models runs roughly $2-$15 per million output tokens depending on vendor and tier, with open-weight API providers lower. So this deployment lands at or below the cheapest API equivalents — but only because it moves 15 billion tokens a month and holds 40% utilization.
Now the counterfactual. Cut volume to 2 billion tokens/month (still a lot) and watch utilization drop to ~8% because the same hardware is mostly idle. Annual TCO barely falls — maybe to $380k, since staffing and colo are fixed — and effective cost becomes ~$16 per million tokens. The identical hardware is now 3-6x more expensive than APIs. Same GPUs, same model, tenfold worse economics. Utilization is not a detail; it is the model.
Break-even rules of thumb
From running this model across many deployments, our rough guidance:
| Monthly output tokens | 40% utilization fleet size | Likely cheaper option |
|---|---|---|
| < 500M | Fraction of one node | API, decisively |
| 500M - 2B | ~1 node (hard to fill) | API usually, unless sovereignty mandates on-prem |
| 2B - 8B | 1-2 nodes | Close call; on-prem wins with existing infra team |
| > 8B | 2+ nodes, batching well | On-prem, by 2-5x |
Failure modes we have actually seen
- The idle cluster. Bought for a use case that shipped six months late; utilization sat near 5% for two quarters. The TCO model should include a ramp curve, not day-one volume.
- The unstaffed deployment. Hardware amortized, model running, and one departing engineer was the entire bus factor. If you cannot fund the 0.3+ FTE, do not buy the GPUs.
- The refresh surprise. Node specified with no VRAM headroom; the model the team actually wanted eight months later required a different quantization tier and quality suffered.
- Utilization theater. Metrics showed high batch occupancy during business hours; monthly tokens told the real story. Always integrate tokens over a month, not instantaneous load.
Conclusion
On-prem inference in 2026 is a volume-and-utilization business. The ~55% of enterprise inference now running on-prem got there by crossing thresholds, not by faith. Build the model, measure your real token mix, be brutally honest about utilization and staffing, and let the arithmetic decide — then weigh the sovereignty and latency factors that no spreadsheet captures.
Want help modeling your own deployment? Contact us — we run these TCO analyses and private inference systems for a living, and we will tell you when the API is the right answer. More engineering write-ups are on our blog.
References and further reading
1. OWASP Top 10 for LLM Applications (2025) — relevant for the compliance overhead side of on-prem deployments. https://genai.owasp.org/ 2. vLLM documentation — PagedAttention, continuous batching, and throughput benchmarking methodology. https://docs.vllm.ai/ 3. SGLang documentation — RadixAttention and serving optimizations relevant to utilization math. https://docs.sglang.ai/ 4. EU AI Act, official text and timeline (EUR-Lex / artificialintelligenceact.eu). https://artificialintelligenceact.eu/ 5. NVIDIA performance documentation for H100-class inference (NVDEX / NVIDIA developer docs). https://docs.nvidia.com/ 6. Uptime Institute annual data center survey reports — PUE and facility cost trends. https://uptimeinstitute.com/ 7. Epoch AI — trends in model compute and hardware requirements over time. https://epoch.ai/ 8. SemiAnalysis GPU market analyses — hardware pricing and supply dynamics. https://semianalysis.com/