A regulated financial services client came to us with a hard constraint: no data could leave their network perimeter. Not compressed. Not anonymised. Not routed through a VPC endpoint. Their legal team had reviewed the OpenAI and Anthropic enterprise agreements and determined that the data processing terms were incompatible with their obligations under local banking regulation. The answer was on-premise LLM deployment — and it was nothing like the blog posts described.

This article covers what a real on-premise inference deployment looks like from hardware selection through to operational monitoring. It is written for teams who have received a similar constraint and are trying to figure out what they are actually getting into.

Why Cloud LLM APIs Fail in Regulated Contexts

The obvious answer is data residency: certain jurisdictions require that personal or financial data never crosses a national border, and cloud API calls to US-based inference endpoints violate that requirement regardless of contractual protections. But the constraint is often broader. Air-gapped defence systems cannot reach external APIs at all. Healthcare systems processing patient records may face HIPAA audit requirements that cloud providers cannot satisfy. Industrial control environments have operational technology (OT) networks deliberately isolated from the internet.

Even in cases where the legal constraint could theoretically be satisfied, enterprise security teams often refuse. The attack surface of sending internal prompts to an external service — particularly one that logs interactions for safety monitoring — is simply not acceptable to organisations with mature security programmes. The shift to on-premise is not always about regulation. Sometimes it is about risk appetite.

The practical consequence is that you inherit the entire serving stack: hardware procurement, model management, inference optimisation, monitoring, and upgrade management. Cloud providers have abstracted all of this. On-premise deployment means building it yourself, or accepting a vendor-managed appliance that restricts your flexibility.

 

Common On-Premise Triggers

  • Data residency regulations (GDPR, PDPA, local banking law)
  • Air-gapped networks with no internet access
  • Healthcare data governed by HIPAA or equivalent
  • Enterprise security policy refusing external data processing
  • Defence and government classified environments

Hardware Selection

The first decision is whether to use GPU servers, CPU-only hardware, or purpose-built inference appliances. For most production deployments, NVIDIA A-series or H-series GPUs are the practical choice: A10G for cost-sensitive deployments, A100 or H100 for high-throughput or large-model requirements. The GPU memory determines which models you can run at which quantisation levels — a 24 GB card running a 7B model at 4-bit quantisation delivers acceptable quality for many tasks, but a 70B model at the same quality requires four such cards minimum.

CPU-only inference is viable for smaller models (7B and below) on modern server CPUs using frameworks like llama.cpp. Throughput is lower but the hardware cost is dramatically reduced, and many regulated environments already have spare server capacity. For workloads with bursty demand rather than continuous load, CPU inference often makes more economic sense than acquiring and maintaining GPU infrastructure.

Network storage matters more than it sounds. Model weights for a 70B model exceed 35 GB at 4-bit quantisation. Loading from a slow NAS on every pod restart adds minutes to recovery time. Local NVMe SSDs for model weights, with NAS as backup, is the correct architecture for production.

Model Choice and Quantisation

Open-weight models — Llama 3, Mistral, Qwen, Phi — are the practical foundation for on-premise deployment. They can be downloaded, stored, quantised, and served without ongoing API costs or terms-of-service constraints. The choice between them depends on the task profile: Llama 3 70B Instruct performs well on complex reasoning tasks, Phi-3 Mini performs surprisingly well on structured extraction tasks at a fraction of the resource requirement.

Quantisation converts model weights from 16-bit or 32-bit floating point to 4-bit or 8-bit integer representations. A 70B model that requires approximately 140 GB of VRAM at full precision fits on a 4× A10G setup at 4-bit quantisation. Quality degradation from 4-bit quantisation is measurable but acceptable for most enterprise tasks. For tasks where accuracy is critical — legal document analysis, medical note summarisation — run 8-bit quantisation and accept the hardware cost.

GGUF format models via llama.cpp offer the best flexibility for CPU and mixed CPU/GPU inference. GPTQ or AWQ quantised models via vLLM or TGI are better for pure GPU serving. The choice of format should follow the serving framework, not the other way around.

Serving Infrastructure

vLLM is currently the most production-ready open-source inference server for GPU deployments. It implements PagedAttention for efficient KV-cache management, supports continuous batching to maximise GPU utilisation under variable load, and exposes an OpenAI-compatible API surface that allows existing integrations to switch to on-premise deployment with minimal code changes. Text Generation Inference (TGI) from Hugging Face is a strong alternative with similar capabilities.

The serving layer needs load balancing, health checks, and graceful restart handling. A single inference server process crashing under load should not take down the entire service. Kubernetes with GPU operator handles pod scheduling and restart management; for simpler environments, a reverse proxy in front of two serving replicas provides adequate resilience.

Prompt caching matters for interactive use cases. vLLM implements prefix caching that reuses KV-cache entries for repeated prompt prefixes — valuable for systems where most requests start with a long system prompt. In our financial services deployment, enabling prefix caching reduced P90 latency by 40% and GPU memory pressure by 25%.

 

Serving Stack Decision Tree

  • GPU cluster: vLLM with PagedAttention + Kubernetes GPU operator
  • CPU only: llama.cpp server or Ollama for internal tooling
  • Mixed CPU/GPU: llama.cpp with GPU offload layers configured
  • High availability: nginx upstream to 2+ serving replicas
  • API compatibility: OpenAI-compatible endpoint to reuse existing integrations

Operational Reality

The part that surprises teams coming from cloud APIs is the operational burden. Model updates require downloading new weights, validating them, staging to a test replica, and promoting — a process that takes hours, not minutes. Security patches to the inference framework require the same treatment. GPU hardware fails: memory errors, thermal throttling, and driver incompatibilities are real events in multi-year deployments.

Monitoring on-premise inference requires instrumentation that cloud providers handle automatically. You need token throughput, queue depth, P50/P95/P99 latency by prompt length, GPU utilisation, and error rates — all feeding into alerting. Prometheus + Grafana is the standard open-source stack. vLLM and TGI both expose Prometheus metrics natively.

The net result is that on-premise LLM deployment costs more than cloud APIs for low-volume workloads. The break-even depends heavily on your inference volume, but for any organisation running more than a few thousand requests per day continuously, the economics typically favour on-premise within 12–18 months of hardware amortisation. The regulatory constraint is rarely optional — but the economics often end up being a second, independent justification.