Local or Cloud? A Decision Framework and Cost Model for Routing LLM Traffic in 2026
Two years ago, the local-versus-cloud question was ideological. Teams either drank the self-hosting Kool-Aid or wrote everything against a frontier API and moved on. In 2026, that question is mostly answered, and the answer is boring in the best way: both, behind a router. Hybrid local-plus-cloud architectures are now the mainstream enterprise pattern — DreamFactory's industry tracking put on-prem inference at roughly 12 percent of deployments in 2023, rising toward a projected 55 percent-plus share by 2026, and virtually none of those on-prem deployments are hermetically sealed. They all keep a cloud path.
We run hybrid LLM infrastructure for clients across legal, healthcare, finance, and logistics. This article is the decision framework and cost model we actually use, written down for the first time in one place.
Why single-mode infrastructure fails
Start with the failure modes, because they motivate everything else.
All-cloud fails on: cost at scale (agent fan-out multiplies token spend), data governance (regulators and counsel get serious about what leaves the perimeter), latency-sensitive loops, vendor outages that take your whole product down, and quota/rate-limit ceilings during bursts.
All-local fails on: capital utilization (idle GPUs are pure loss), capability ceilings (your 8B model cannot do everything), burst absorption (Black Friday does not care about your fixed capacity), and operational burden on teams without inference-ops experience.
The hybrid pattern addresses each failure with the other mode's strength. The engineering problem is deciding, per request, which mode to use — and doing it cheaply, reliably, and auditably. That is routing.
The four-axis decision framework
Every LLM request in an enterprise system can be scored on four axes. We run this assessment with clients as a structured workshop; you can do a first pass in an afternoon.
Axis 1: Data sensitivity
Classify what the prompt and (especially) the retrieved context contain:
| Tier | Data examples | Constraint |
|---|---|---|
| S0 | Public docs, synthetic data, general knowledge Q&A | Anything goes, optimize for cost |
| S1 | Internal policy, non-sensitive telemetry, aggregate analytics | Private VPC endpoint or contractually-protected API tier |
| S2 | PII, customer content, internal code, contracts | Local inference, or cloud only under DPA + zero-retention terms |
| S3 | Regulated data (PHI, PCI, classified, M&A material) | Local only, full audit trail, often air-gapped |
Axis 2: Task complexity
Score the cognitive difficulty of the call, not the importance of the call. Classification, extraction, summarization of bounded documents, schema-bound generation: low complexity, an 8B-class SLM is at parity. Open-ended reasoning, novel code synthesis, ambiguous multi-constraint planning: high complexity, frontier territory. (We covered the benchmark evidence in our SLM agents guide — see /blog.)
Axis 3: Latency profile
Interactive (user waiting, sub-second budget for first token), trajectory (inside an agent loop, latency compounds across calls), or batch (overnight, latency irrelevant). Interactive and trajectory calls favor local small models on first-token latency; batch calls favor whatever is cheapest, including spot-priced cloud capacity.
Axis 4: Volume and duty cycle
This axis decides infrastructure, not per-request routing. Sustained, predictable volume amortizes GPUs; spiky or low volume favors consumption pricing.
Putting it together
The routing table we configure by default:
| Request profile | Sensitivity | Complexity | Route |
|---|---|---|---|
| Document classification | S2 | Low | Local 8B |
| Customer-facing draft generation | S1-S2 | Medium | Local 8-14B + verifier, escalate on flags |
| Code generation over internal repo | S2 | High | Local 70B if capacity, else zero-retention cloud tier |
| Market research synthesis from public web | S0 | High | Frontier cloud API |
| Batch backfill over archived data | S2 | Low | Local, scheduled at off-peak |
| Burst overflow, any tier | S0-S1 | Any | Cloud, capped by budget guardrails |
The cost model (with real numbers)
Here is the spreadsheet we build for every engagement. Inputs: requests per day, average input/output tokens, concurrency profile, and cloud price sheet.
Local side. A single L4-class GPU (24GB), owned, amortized over three years with power, cooling, and a fraction of an ops engineer: call it $450-700/month all-in. Serving an 8B model with vLLM at FP8/AWQ quantization, it sustains on the order of 2,000-4,000 aggregate output tokens/second under continuous batching at moderate concurrency. At 40 percent average utilization that is roughly 100-200M output tokens/month, or $0.25-0.70 per million output tokens. Add a second GPU for redundancy and the per-token number roughly doubles but your availability SLA becomes real.
Cloud side. 2026 mid-tier API models run roughly $0.10-0.60 per million input tokens and $0.40-3.00 per million output tokens; flagships run $1-10+ per million input and $4-15+ per million output.
Break-even math. Take a workload of 100k requests/day at 2,000 input / 500 output tokens each (200M in / 50M out monthly). Against a mid-tier cloud model at $0.25/M in and $1.00/M out, API spend is about $95,000/year. The same volume on two owned GPUs at $1,200/month total is about $14,400/year — a ~6.5x reduction, consistent with the 5-15x range we typically achieve for production agentic workloads. The "10-100x" headline numbers assume higher utilization, owned-hardware amortization, and comparison against flagship pricing. Below roughly 5-20k requests/day (depending on token shape), cloud wins outright because GPU idle time dominates. Always model your own numbers; the crossover is unforgiving.
One more line item people forget: egress and embedding costs. RAG workloads re-embed on update cycles, and hybrid systems that sync knowledge between local and cloud stores generate transfer costs and complexity. Budget for it.
The routing architecture
The stack we deploy, all standard components:
1. Gateway/router. LiteLLM Proxy, or a thin custom service. Holds backend configs, keys, budgets, and routing rules. Stateless, horizontally scaled, behind your load balancer. Exposes an OpenAI-compatible endpoint so application code never knows where inference happens. 2. Policy engine. Sensitivity tiers and complexity scores map to backend allowlists. Enforcement belongs in the router, not in application code — developers will forget, and the router is the one choke point you can audit. 3. Local inference tier. vLLM or TensorRT-LLM, one model pool per model class, health-checked, with queued bursting to cloud when local utilization crosses a threshold (we typically trigger at 70-80 percent). 4. Cloud tier. Multiple providers, both for capability variety and outage insurance. Zero-retention contract terms for anything above S0. 5. Observability. Per-request logs of route chosen, reason, latency, tokens, cost — plus weekly cost-per-request-class reporting. If you cannot answer "what did classification traffic cost us last week and where did it run," the router is not finished.
A concrete routing rule set
Expressed as pseudocode (indentation, not a real language):
route(request): if request.sensitivity in [S3]: return local_only if request.sensitivity == S2 and local.capacity_ok: return local if request.complexity == high: return cloud_frontier # allowed since sensitivity <= S1 if local.utilization < 0.8 and request.latency_class != batch: return local if request.budget_remaining > cost_estimate(request, cloud): return cloud_mid_tier return queue_for_local_offpeak
Failover: any backend error or 2x latency regression retries once elsewhere, with the router stamping the request so downstream debugging does not lie awake wondering why answers differ under load.
Failure modes we have learned the hard way
- Untested failover paths. Every router has a cloud fallback config; few teams have ever actually triggered it deliberately. Ours get chaos-tested monthly. The first time you exercise failover should not be during a cloud-region incident.
- Router drift. Business logic creeps into routing rules until nobody can explain why S1 traffic goes to three different backends. Review routing configs like code, with change control.
- The anonymization illusion. Covered above; it remains the most common compliance gap we find.
- Cost guardrail surprises. Burst-to-cloud without budget caps turns a traffic spike into a five-figure invoice. Cap by request class, alert at thresholds, and make the degraded path (queue locally) acceptable by design.
- Model version skew. Local and cloud backends update on different schedules; agent behavior subtly shifts. Pin versions, eval on every change, and canary new versions on a traffic slice before fleet rollout.
When to skip the hybrid pattern
Being honest about the alternative: a two-person startup with an internal tool and no sensitive data should just use one cloud provider. A hardened on-prem-only deployment with zero internet egress (some defense and clinical contexts) should not maintain a cloud path just for symmetry — the compliance surface is not worth it. Hybrid is the right default for the broad middle: organizations with meaningful volume, some sensitive data, and quality-critical workloads. That describes most of the enterprises we work with.
Conclusions
Route on sensitivity, complexity, latency, and volume — in that order of veto power. Expect 60-85 percent of volume to land locally once you classify honestly, a 5-15x cost reduction versus naive all-cloud at production scale, and frontier cloud models carrying the tasks that genuinely need them. Invest in the router as a first-class service with observability, budgets, and tested failover, because in a hybrid system the router is your reliability boundary.
References and further reading
1. Scholtens, A., et al. — "SLMs are the Future of Agentic AI" (NVIDIA position paper), arXiv:2506.02153 — https://arxiv.org/abs/2506.02153 2. LiteLLM Proxy documentation (routing, fallbacks, budgets) — https://docs.litellm.ai 3. vLLM: Kwon, W., et al. — "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023 — https://arxiv.org/abs/2309.06180 4. DreamFactory Software — API and AI industry statistics (on-prem inference deployment trends) — https://dreamfactory.io 5. OpenAI API pricing documentation — https://platform.openai.com/docs/pricing 6. DeepSeek API pricing and model documentation — https://api-docs.deepseek.com 7. Martínez, R., et al. — "RouteLLM: Learning to Route LLMs with Preference Data," arXiv:2406.18665 — https://arxiv.org/abs/2406.18665 8. Ding, D., et al. — "Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing," arXiv:2404.14618 — https://arxiv.org/abs/2404.14618
Want help designing (or rescuing) a hybrid LLM routing setup for your workloads? Contact us and we will run the four-axis assessment and cost model against your real traffic. More studio engineering write-ups are on the blog.