A practical stack for local LLM inference in 2026

Introduction

The deployment of large language models on private infrastructure has evolved from experimental prototypes to engineered production systems. Organizations in regulated sectors—including finance, healthcare, and manufacturing—are evaluating and adopting private LLM infrastructure driven by data sovereignty requirements, cost predictability considerations, and latency constraints that centralized cloud APIs cannot address.

This article presents an engineering analysis of common architectural patterns used in private AI deployments. We examine the technical decisions at each infrastructure layer, provide practical guidance based on modern open-source tooling and industry approaches, and outline the evaluation frameworks necessary for production adoption.

The stack architecture presented here reflects established patterns in the open-source community and provides technical reference points for CTOs, AI engineers, and infrastructure teams evaluating private LLM systems.

The Stack: Decision framework

A typical production stack consists of three layers: runtime, quantization, and retrieval. Each layer presents 2-3 viable options depending on workload characteristics and resource constraints.

Layer 1: Runtime selection

Two primary runtimes dominate production deployments based on access patterns and available resources:

llama.cpp — for single-user and resource-constrained scenarios

llama.cpp is suitable when memory efficiency, portability, and local execution are prioritized. It is commonly selected for workstation, edge, and privacy-sensitive deployments where inference must occur on consumer-grade hardware or in environments without datacenter GPU infrastructure.

When to choose:

• Single-user or low-concurrency workloads (<5 concurrent users).

• CPU-only or consumer-grade hardware.

• Inference on edge devices or workstations.

• Memory constraints (<16GB VRAM).

Configuration example:

4-bit quantized Llama 3.1 8B on 16GB RAM
./llama-server \
  --model llama-3.1-8b-instruct.Q4_K_M.gguf \
  --ctx-size 8192 \
  --n-gpu-layers 32 \
  --threads 8 \
  --port 8080

Performance characteristics:

• Cold start: ~2-5 seconds.

• Token throughput: 15-40 tokens/sec (depends on hardware).

• Memory overhead: Minimal (1-2GB beyond model size).

vLLM — for multi-user serving and GPU acceleration

vLLM is selected when high-throughput multi-user serving is required and GPU infrastructure is available. Its PagedAttention mechanism and continuous batching support make it appropriate for concurrent workloads at scale.

When to choose:

• Multi-user serving (>10 concurrent users).

• GPU infrastructure available (NVIDIA A100, H100, or equivalent).

• High-throughput requirements (>100 req/min).

• Batching and continuous batching needed.

Configuration example:

vLLM server with Llama 3.1 70B
from vllm import LLM, SamplingParams
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,  # 4x A100 GPUs
    max_model_len=8192,
    trust_remote_code=True
)
Batched inference
prompts = ["Summarize...", "Translate...", "Analyze..."]
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(prompts, sampling_params)

Performance characteristics:

• Cold start: 30-90 seconds (model loading).

• Token throughput: 100-500 tokens/sec (with batching).

• Memory efficiency: PagedAttention reduces waste by 40%.

Real-world data:

Metric	llama.cpp (CPU)	llama.cpp (GPU)	vLLM (4xA100)
Concurrent users	1-5	5-15	50-200
Latency (p95)	2.5s	800ms	400ms
Cost ($/hour)	$0.20	$1.50	$12.00
Setup complexity	Low	Low	Medium

Performance values in this article are engineering references. Actual results depend on hardware configuration, model architecture, workload characteristics, optimization techniques, and deployment environment.

Layer 2: Quantization strategy

Quantization represents the compression challenge — reducing model size while preserving capability. GGUF format (from the llama.cpp ecosystem) has emerged as the de facto standard for its flexibility and mature tooling.

Quantization levels explained

Q4_K_M — The pragmatic default

• Size: 4.1 bits per weight (e.g., 8B model = 4.9GB).

• Accuracy: 95-98% of FP16 quality.

• Use case: General-purpose deployment.

• Sweet spot for 8B-70B models.

Q5_K_M — When precision matters

• Size: 5.1 bits per weight (e.g., 8B model = 5.9GB).

• Accuracy: 98-99% of FP16 quality.

• Use case: Coding, math, structured outputs.

• Recommended for 13B+ models.

Q8_0 — Near-lossless compression

• Size: 8.5 bits per weight (e.g., 8B model = 9.2GB).

• Accuracy: 99%+ of FP16 quality.

• Use case: Mission-critical applications.

• Only when memory allows.

Q3_K_M — Aggressive compression (use cautiously)

• Size: 3.3 bits per weight (e.g., 8B model = 3.8GB).

• Accuracy: 85-92% of FP16 quality.

• Use case: Edge deployment, rapid prototyping.

• Expect quality degradation.

Quantization workflow

Convert HuggingFace model to GGUF
python convert.py --outfile llama-3.1-8b.fp16.gguf meta-llama/Meta-Llama-3.1-8B-Instruct
Quantize to Q4_K_M
./quantize llama-3.1-8b.fp16.gguf llama-3.1-8b.Q4_K_M.gguf Q4_K_M
Verify quantization quality
./perplexity --model llama-3.1-8b.Q4_K_M.gguf --file test-corpus.txt

Quality benchmarks (Llama 3.1 8B on MMLU):

Quantization	MMLU Score	File Size	Memory Usage
FP16	69.4%	16.0GB	18GB
Q8_0	69.1%	9.2GB	11GB
Q5_K_M	68.7%	5.9GB	8GB
Q4_K_M	68.2%	4.9GB	7GB
Q3_K_M	65.8%	3.8GB	6GB

Performance values in this article are engineering references. Actual results depend on hardware configuration, model architecture, workload characteristics, optimization techniques, and deployment environment.

Layer 3: Retrieval-augmented generation (RAG)

When LLMs require external knowledge access, a three-component retrieval architecture typically emerges:

Component 1: Vector database

Qdrant — widely adopted for mid-scale deployments

Qdrant is selected when deployment simplicity and moderate scale (<10M vectors) are sufficient. It provides HNSW indexing with manageable operational overhead.

docker-compose.yml
services:
  qdrant:
    image: qdrant/qdrant:v1.9.0
    ports:
      - "6333:6333"
    volumes:
      - ./qdrant_storage:/qdrant/storage
    environment:
      QDRANT__SERVICE__GRPC_PORT: 6334

Milvus — for scale >10M vectors

Milvus is chosen when horizontal scaling and distributed architecture are required. It handles larger vector corpora at the cost of increased operational complexity.

• Better horizontal scaling.

• Distributed architecture.

• Higher operational complexity.

Configuration example:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
client = QdrantClient(host="localhost", port=6333)
Create collection with HNSW indexing
client.create_collection(
    collection_name="knowledge_base",
    vectors_config=VectorParams(
        size=1024,  # BGE-M3 dimensionality
        distance=Distance.COSINE
    ),
    hnsw_config={
        "m": 16,              # Number of edges per node
        "ef_construct": 100   # Construction quality
    }
)

Component 2: Embedding model

BGE-M3 — multilingual and multi-representation

BGE-M3 is selected when multilingual support and multiple retrieval modalities are needed. It provides dense vectors for semantic similarity, sparse vectors for lexical matching, and ColBERT mode for token-level precision.

• Dense vectors (1024-dim) for semantic similarity.

• Sparse vectors for lexical matching.

• ColBERT mode for token-level precision.

Deployment:

from FlagEmbedding import BGEM3FlagModel
model = BGEM3FlagModel('BAAI/bge-m3', use_fp16=True)
Generate embeddings
texts = ["Document 1...", "Document 2..."]
embeddings = model.encode(
    texts,
    batch_size=32,
    max_length=8192,
    return_dense=True,
    return_sparse=False,
    return_colbert_vecs=False
)['dense_vecs']

Performance:

• Embedding speed: 40ms per document (CPU).

• 15ms per document (GPU).

• Multilingual support: 100+ languages.

Component 3: Reranker (optional)

Precision-critical applications often benefit from cross-encoder reranking:

from sentence_transformers import CrossEncoder
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3')
Initial retrieval: top 20 candidates
candidates = vector_db.search(query, limit=20)
Rerank to top 5
scores = reranker.predict([(query, doc) for doc in candidates])
top_docs = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:5]

Tradeoff: +150ms latency for +8% recall improvement

Production deployment architecture

A reference architecture supporting 50-user organizations typically looks like this:

                    ┌─────────────────┐
                    │   Nginx/Caddy   │
                    │  (Load Balancer) │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │   API Gateway   │
                    │  (Auth, Limits)  │
                    └────────┬────────┘
                             │
            ┌────────────────┼────────────────┐
            │                │                │
    ┌───────▼──────┐ ┌──────▼───────┐ ┌─────▼──────┐
    │ vLLM Server  │ │ vLLM Server  │ │ vLLM Server│
    │  (Primary)   │ │  (Replica 1) │ │ (Replica 2)│
    └───────┬──────┘ └──────┬───────┘ └─────┬──────┘
            │                │                │
            └────────────────┼────────────────┘
                             │
                    ┌────────▼────────┐
                    │     Qdrant      │
                    │  (Vector Store)  │
                    └─────────────────┘

Infrastructure requirements:

• Compute: 3x servers with NVIDIA A100 (40GB) or equivalent.

• Storage: 500GB NVMe SSD per node.

• Network: 10Gbps internal, 1Gbps external.

• Memory: 128GB RAM per node.

Configuration files:

k8s/vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-3-70b
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - --model=meta-llama/Meta-Llama-3.1-70B-Instruct
          - --tensor-parallel-size=2
          - --max-model-len=8192
          - --trust-remote-code
        resources:
          limits:
            nvidia.com/gpu: 2
            memory: 128Gi
          requests:
            nvidia.com/gpu: 2
            memory: 128Gi
        ports:
        - containerPort: 8000

Monitoring and observability

Metrics we track

Latency metrics:

Prometheus metrics
inference_latency_seconds.observe(duration)
tokens_per_second.set(tps)
queue_depth.set(waiting_requests)

GPU utilization:

NVIDIA DCGM exporter
nvidia_gpu_utilization{gpu="0"} 87
nvidia_memory_used_bytes{gpu="0"} 34359738368
nvidia_temperature_celsius{gpu="0"} 72

Dashboard configuration (Grafana):

• Request latency (p50, p90, p95, p99).

• Token throughput (tokens/sec).

• GPU utilization (per device).

• Memory usage (VRAM and system RAM).

• Queue depth and rejection rate.

• Cost per 1M tokens.

Alerting thresholds

alerts.yaml
groups:
name: llm-inference
  rules:
  - alert: HighLatency
    expr: histogram_quantile(0.95, inference_latency_seconds) > 2.0
    for: 5m
    annotations:
      summary: "P95 latency above 2s for 5 minutes"
  - alert: GPUMemoryPressure
    expr: nvidia_memory_used_bytes / nvidia_memory_total_bytes > 0.95
    for: 2m
    annotations:
      summary: "GPU memory usage above 95%"

Cost analysis

Hardware investment (one-time):

• 3x servers with 2xA100 (40GB): $45,000.

• Networking and infrastructure: $5,000.

• Total: $50,000.

Operating costs (monthly):

• Power (3kW @ $0.12/kWh, 730h): $263.

• Cooling and facilities: $150.

• Network bandwidth: $200.

• Maintenance and support: $500.

• Total: $1,113/month.

Break-even analysis:

• OpenAI GPT-4 cost: $0.03/1K input tokens, $0.06/1K output tokens.

• Average request: 1K input + 500 output = $0.06.

• Monthly usage at break-even: ~18,500 requests/month.

• Daily: ~620 requests/day.

For organizations processing >1,000 requests/day, local deployment is cost-effective within 6-12 months.

Security considerations

Network isolation

iptables rules — restrict access to internal network
iptables -A INPUT -p tcp --dport 8000 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 8000 -j DROP

Access control

API gateway with JWT authentication
@app.before_request
def verify_token():
    token = request.headers.get('Authorization')
    if not token or not verify_jwt(token):
        return jsonify({"error": "Unauthorized"}), 401

Model verification

Verify model checksums before deployment
sha256sum llama-3.1-70b-instruct.Q4_K_M.gguf
Compare with official hash from model card

Operational playbook

Deployment checklist

• [ ] Hardware validated (GPU drivers, CUDA version)

• [ ] Models downloaded and verified (checksums match)

• [ ] Quantization quality tested (perplexity within threshold)

• [ ] Runtime configured (context length, batch size)

• [ ] Monitoring enabled (metrics, logs, alerts)

• [ ] Load testing completed (sustained load for 1 hour)

• [ ] Failover tested (replica promotion works)

• [ ] Backup and recovery procedures documented

Common issues and fixes

Issue: High latency spikes

• Cause: GPU memory fragmentation.

• Fix: Restart vLLM server, consider reducing batch size.

Issue: Out-of-memory errors

• Cause: Context length too large for available VRAM.

• Fix: Reduce --max-model-len or use aggressive quantization.

Issue: Poor response quality

• Cause: Over-aggressive quantization.

• Fix: Test with Q5_K_M or Q8_0 quantization.

Evaluation before deployment

Production AI systems should be evaluated systematically before adoption. Model selection and system architecture should be driven by measurable performance rather than assumptions.

Evaluation dimensions

Rigorous pre-deployment evaluation typically covers four primary dimensions:

Retrieval quality

When RAG systems are involved, retrieval quality directly impacts generation accuracy:

• Recall@K: Percentage of relevant documents retrieved in top K results (typically K=5, 10, 20).

• Precision: Proportion of retrieved documents that are relevant.

• MRR (Mean Reciprocal Rank): Average reciprocal position of the first relevant document.

Target thresholds (domain-dependent):

• General knowledge: Recall@10 >80%.

• Domain-specific (legal, medical): Recall@10 >90%.

Generation quality

LLM output quality requires evaluation on multiple criteria:

• Answer correctness: Factual accuracy against ground truth.

• Citation accuracy: Correct attribution to source documents.

• Hallucination rate: Percentage of generated content not supported by context.

• Relevance: Response alignment with query intent.

Evaluation approaches:

• Human evaluation (gold standard, expensive).

• LLM-as-judge (automated, requires validation).

• Rule-based heuristics (fast, limited scope).

System performance

Performance characteristics must align with production requirements:

• Latency distribution: p50, p95, p99 response times under load.

• Throughput: Requests per second at target latency.

• GPU utilization: Efficiency of compute resource usage.

• Memory efficiency: VRAM usage patterns and overhead.

Common targets:

• Interactive chat: p95 <300ms.

• Batch processing: Throughput >100 req/min.

• GPU utilization: >70% under load.

Operational reliability

Production systems require reliability engineering:

• Failure handling: Behavior under edge cases (malformed input, context overflow).

• Monitoring coverage: Metrics, logging, alerting completeness.

• Regression testing: Automated validation of quality and performance.

• Graceful degradation: System behavior when components fail.

Evaluation workflow

Example evaluation pipeline structure
def evaluate_llm_system(model, test_dataset):
    results = {
        'retrieval': evaluate_retrieval(model.retriever, test_dataset),
        'generation': evaluate_generation(model.generator, test_dataset),
        'latency': benchmark_latency(model, test_dataset),
        'quality': measure_quality(model, test_dataset)
    }
    # Check against thresholds
    passes_criteria = (
        results['retrieval']['recall@10'] > 0.80 and
        results['latency']['p95'] < 0.5 and
        results['quality']['hallucination_rate'] < 0.05
    )
    return results, passes_criteria

Benchmarking quantization impact

Before deploying quantized models, validate quality degradation is acceptable:

Measure perplexity on evaluation corpus
./perplexity --model llama-3.1-8b.Q4_K_M.gguf --file eval_corpus.txt
Compare against FP16 baseline
Accept quantization if perplexity delta <5%

Evaluation is not a one-time gate. Continuous evaluation in production enables detection of model drift, data distribution shifts, and system degradation.

Production considerations

Deploying an LLM system requires engineering around the model, not only model selection. Production-grade systems incorporate version management, security controls, monitoring infrastructure, and operational procedures.

Model version management

Track model versions, configurations, and artifacts:

model-registry.yaml
models:
  - name: llama-3.1-8b-instruct
    version: v1.2.0
    quantization: Q4_K_M
    sha256: a3f4d9e2b1c8...
    deployment_date: 2026-06-15
    evaluation_metrics:
      recall@10: 0.84
      p95_latency: 420ms
      mmlu_score: 68.2

Maintain lineage from base model through quantization to deployment artifact.

Dataset and document versioning

RAG systems depend on document corpora. Version and track:

• Document corpus snapshots.

• Embedding generation timestamps.

• Indexing configurations.

• Preprocessing and chunking logic.

Document corpus versioning
corpus_metadata = {
    'version': 'v2.3.0',
    'documents_count': 45231,
    'last_updated': '2026-06-20',
    'embedding_model': 'bge-m3',
    'chunking_strategy': 'semantic-512'
}

Evaluation pipelines

Automate regression testing on each deployment:

ci-cd-pipeline.yaml
evaluation:
  - name: retrieval-quality
    test_set: golden_queries_v3.jsonl
    metrics: [recall@10, mrr, precision@5]
    thresholds:
      recall@10: 0.80
  - name: latency-benchmark
    load: 100_req_per_min
    duration: 10_minutes
    thresholds:
      p95_latency: 500ms
  - name: generation-quality
    evaluator: llm_judge
    criteria: [correctness, relevance, citation]

Monitoring

Instrument systems for observability:

• Request-level traces: Track full request lifecycle.

• Model metrics: Token counts, batch sizes, cache hit rates.

• Resource utilization: GPU, CPU, memory, disk I/O.

• Business metrics: User satisfaction, task completion rates.

Structured logging
@trace_request
def handle_inference(request):
    with metrics.timer('inference_latency'):
        result = model.generate(request.prompt)
    metrics.increment('requests_total')
    metrics.histogram('tokens_generated', len(result.tokens))
    return result

Security controls

Private LLM deployments require security architecture:

• Network isolation: Restrict model access to authorized networks.

• Authentication: API keys, JWT tokens, OAuth integration.

• Authorization: Role-based access control (RBAC).

• Input validation: Sanitize prompts, enforce length limits.

• Audit logging: Track all inference requests and responses.

• Model verification: Validate model checksums on deployment.

API security middleware
@require_auth
@rate_limit(requests_per_minute=100)
@validate_input(max_length=4096)
def inference_endpoint(request):
    audit_log.record(user=request.user, prompt=request.prompt)
    return model.generate(request.prompt)

Backup and recovery

Plan for failure scenarios:

• Model artifacts: Backup quantized models, configurations.

• Vector databases: Regular snapshots of indexed documents.

• Monitoring data: Retain metrics for incident analysis.

• Recovery procedures: Documented restoration steps.

Recovery Time Objective (RTO): <30 minutes for model endpoint restoration Recovery Point Objective (RPO): <24 hours for document corpus

Access control

Implement least-privilege access:

rbac-policy.yaml
roles:
  - name: inference_user
    permissions:
      - llm:generate
      - retrieval:query
  - name: admin
    permissions:
      - llm:deploy
      - model:update
      - config:modify

Production LLM systems are infrastructure, not prototypes. They require the same operational discipline as databases, message queues, and other critical components.

Conclusion

Private LLM deployment requires architecture decisions that extend beyond model selection. Successful production systems balance multiple competing concerns: generation quality, retrieval accuracy, response latency, infrastructure cost, security posture, and operational maintainability.

Key engineering principles

The architectural patterns examined in this article converge on several principles:

1. Infrastructure choices are workload-specific: llama.cpp for edge and single-user scenarios; vLLM for high-concurrency multi-user serving

2. Quantization trades memory for quality: Systematic evaluation determines acceptable degradation thresholds

3. Retrieval architecture impacts generation: RAG system design is as critical as model selection

4. Evaluation precedes deployment: Measurable quality gates prevent production incidents

5. Operational discipline is required: Monitoring, versioning, security, and recovery procedures are essential

The role of open-weight models

Open-weight models combined with strong engineering practices enable organizations to build controlled AI systems. The availability of open-source runtimes (llama.cpp, vLLM), standardized formats (GGUF), and robust vector databases (Qdrant, Milvus) has lowered the barrier to private deployment.

However, deployment accessibility should not be confused with deployment simplicity. Production LLM systems require careful architecture, systematic evaluation, comprehensive monitoring, and operational maturity.

Path forward

Organizations evaluating private LLM infrastructure should:

• Define measurable quality and performance requirements before selecting models.

• Prototype with llama.cpp; scale to vLLM as concurrency demands increase.

• Establish evaluation pipelines that run on every deployment.

• Instrument systems for observability from the start.

• Build operational runbooks for common failure modes.

Private AI infrastructure is maturing. What was experimental in 2024 has become engineered practice in 2026. Organizations that approach deployment with rigorous evaluation, solid architecture, and operational discipline are building systems that deliver value while maintaining control.

---

Need guidance on private AI infrastructure? Contact us to discuss architecture reviews, deployment patterns, and optimization strategies.