Kubernetes has become the default infrastructure substrate for production ML systems at companies beyond the smallest scale. It is not the easiest platform for ML workloads — GPU scheduling, storage management, and distributed training require significant configuration — but it provides the consistency, observability, and automation that make ML systems maintainable by more than the one engineer who set them up.

This article covers the practical configuration decisions for running ML workloads on Kubernetes: GPU resource management, model serving deployment patterns, training job orchestration, and cost control for GPU infrastructure. The patterns here come from production ML deployments across multiple clients.

Why Kubernetes for ML Workloads

The primary reason to run ML on Kubernetes is consistency across environments. Training, evaluation, and serving run in the same container images, with the same dependency versions, on the same infrastructure abstraction layer. This eliminates the "works on my machine" problem and the environment drift that causes production model serving to behave differently from training. The reproducibility value compounds as teams and models scale.

Resource isolation is the second reason. A Kubernetes cluster running both serving workloads (latency-sensitive, need predictable GPU allocation) and training workloads (throughput-sensitive, can tolerate preemption) can allocate resources correctly to each using priority classes and resource quotas. Without this isolation, a training job that consumes all available GPU memory on a node can delay serving requests — an unacceptable outcome for production systems.

The third reason is the ecosystem. Kubeflow Pipelines, Argo Workflows, Seldon, KServe, and Ray are all Kubernetes-native ML platforms. Adopting Kubernetes early means these tools are available without significant migration cost. Choosing a proprietary ML platform early may lock the team into less flexible infrastructure as requirements grow.

GPU Scheduling in Kubernetes

NVIDIA GPU Operator handles GPU driver management, container runtime configuration, and device plugin installation on Kubernetes nodes. Install it on all nodes with NVIDIA GPUs — it replaces the manual process of installing drivers, NVIDIA Container Toolkit, and the device plugin separately, and keeps them consistent across nodes. Time Sharing with the GPU Operator allows multiple pods to share a single GPU for inference workloads that do not require a full GPU, improving utilisation for lighter serving use cases.

Request GPU resources explicitly in pod specifications using the `nvidia.com/gpu` resource key. A pod that does not request a GPU but attempts to use one will fail unpredictably — either because the CUDA environment is not configured, or because another pod holds the device lock. Make GPU requests explicit and match them to the actual requirement: requesting 1 GPU for a workload that needs 4 is a configuration error that wastes 3 GPUs.

Node affinity and taints should be used to dedicate GPU nodes to GPU workloads and prevent non-GPU pods from consuming node resources that GPU pods need. A taint on GPU nodes (`nvidia.com/gpu:NoSchedule`) ensures only pods that tolerate this taint — your ML serving and training pods — will be scheduled there. This prevents general application pods from fragmenting GPU node capacity.

 

GPU Kubernetes Configuration

  • NVIDIA GPU Operator: manages drivers, runtime, device plugin automatically
  • Resource requests: always explicit nvidia.com/gpu in pod spec
  • Node taints: isolate GPU nodes from general workloads
  • Priority classes: serving > training for GPU preemption
  • Time sharing: for small serving models that share a GPU

Model Serving Deployment Patterns

KServe (formerly KFServing) provides a Kubernetes-native model serving framework with inference service CRDs, canary deployments, automatic scaling, and multi-framework support. It is the most comprehensive open-source option for production model serving on Kubernetes. For simpler use cases, a standard Kubernetes Deployment with a custom serving container (vLLM, TGI, or a custom FastAPI inference server) is often more appropriate — fewer abstractions to configure and debug.

Horizontal Pod Autoscaler (HPA) on CPU metrics does not work well for GPU inference pods — CPU utilisation does not correlate with inference throughput because the inference compute happens on the GPU. Use custom metrics (inference requests per second, queue depth) as the HPA target, exposed via a Prometheus adapter. This allows the cluster to scale serving pods based on actual inference load rather than CPU utilisation.

Rolling deployments for model serving require careful configuration. The `maxUnavailable: 0` rolling update strategy ensures that new pods are running and passing health checks before old pods are terminated, preventing serving gaps during model version transitions. Configure readiness probes that load the model and run a test inference before marking the pod ready — a pod that is running but has not finished loading the model will receive traffic before it can serve correctly.

Training Job Orchestration

Kubernetes Jobs are suitable for simple single-node training runs. For distributed training — data parallelism across multiple GPUs or multiple nodes — Kubeflow Training Operator provides PyTorchJob and TFJob resources that manage the distributed training lifecycle, including coordinating between workers, handling failures, and cleaning up resources after completion.

GPU preemption for training jobs is valuable when serving and training share the same cluster. Training jobs can be marked as lower priority than serving pods, so when cluster capacity is tight, training job pods are preempted to free GPU capacity for serving. Configure checkpointing in training code to save state periodically — preempted jobs should resume from the last checkpoint, not restart from the beginning. Without checkpointing, preemption can destroy hours of training progress.

Spot instances (AWS) or preemptible VMs (GCP) for training nodes reduce compute costs by 60–80% compared to on-demand GPU instances. The operational cost is that nodes can be reclaimed with short notice — typically 2 minutes. This is compatible with training workloads that checkpoint regularly but requires careful cluster configuration to prevent preemption events from affecting serving pods on separate on-demand nodes.

Cost Management for GPU Infrastructure

GPU utilisation tracking is the foundation of cost management. A cluster with 20% average GPU utilisation is spending 80% of its GPU budget inefficiently. Track GPU utilisation per workload using DCGM metrics exposed via NVIDIA DCGM Exporter to Prometheus. Identify low-utilisation serving pods (candidate for smaller GPU instances or time sharing), idle training nodes (candidate for cluster autoscaler scale-down), and workloads that are requesting more GPUs than they are using.

Cluster autoscaler with GPU node pools reduces cost during low-traffic periods by scaling down to zero GPU nodes when no GPU workloads are running. Configure the autoscaler with a minimum of zero nodes for GPU node pools and a scale-down delay that prevents rapid scale-up/scale-down oscillation. Accept the cold-start latency of scaling up a GPU node when on-demand serving is required, or maintain a minimum of one on-demand GPU node for low-latency serving with spot instances for burst capacity.

Namespace resource quotas prevent any single team or project from consuming disproportionate cluster resources. Set GPU quotas per namespace that reflect the allocation you have planned for each project, and require quota increase requests to go through an approval process. Without quotas, well-intentioned engineers will request the largest GPU they can get for a task that would work fine on a smaller one — a common pattern in research-oriented teams that compounds into significant unnecessary cost.