Software CI/CD pipelines are well-understood. You push code, the pipeline runs tests, and the build either passes or fails. ML pipelines are more complex: the artifact you are deploying is not just code — it is a trained model with weights that encode learned patterns from training data. The same code with different training data produces a different artifact. Data drift makes a model that passed all tests six months ago produce subtly wrong outputs today. Retraining on new data can regress performance on old scenarios.

This article describes how to structure a CI/CD pipeline that handles the full ML model lifecycle: from data validation through training, testing, staged deployment, and post-deployment monitoring. The patterns here come from production MLOps work across several client systems where model degradation had real business consequences.

Why ML CI/CD Is Different from Software CI/CD

In software CI/CD, the test suite is deterministic: given the same code, the same tests produce the same result. ML pipelines have three sources of non-determinism. Training randomness means two runs with the same code and data may produce models with slightly different weights. Data changes mean the same code trained on new data may perform differently on old validation sets. Model behaviour changes non-linearly — a model can simultaneously improve on aggregate metrics and regress badly on a specific important subgroup.

The consequence is that ML CI/CD requires three parallel tracks of testing that software CI/CD does not: data validation (is the training data distribution consistent with expectations?), model quality testing (does this model meet minimum performance thresholds on held-out evaluation sets?), and regression testing (does this model maintain performance on previously validated scenarios?). Each track can fail independently and should gate deployment independently.

Retraining frequency adds another dimension. Many teams retrain on a schedule — weekly, monthly — rather than on every code change. The CI/CD pipeline must handle triggered retraining runs, evaluate the resulting model against the current production model, and either promote automatically or escalate for human review depending on the magnitude of the change.

Model Registry and Versioning

A model registry is the central store for trained model artifacts, their associated metadata, and their lifecycle state. Every trained model should be registered with: the training data version used, the code version (git commit), all hyperparameters, evaluation metrics on the validation set, and the timestamp. This makes every production model reproducible and auditable.

MLflow, Weights & Biases, and DVC are the common open-source options. Managed options include SageMaker Model Registry and Vertex AI Model Registry. The choice depends on your broader infrastructure — teams already using MLflow for experiment tracking should use MLflow Model Registry; teams with heavy GCP investment should use Vertex. The key capability is lifecycle state management: a model should move through Draft → Staging → Production → Archived states with human approval gates at the Staging → Production transition.

Model versioning is different from code versioning. You need to retain production models for at least the duration of your rollback window plus any regulatory audit requirements. A model that was in production for six months may need to be re-runnable eighteen months later for a compliance audit. Version your model artifacts the same way you version your code releases — never delete a model that was ever in production.

 

Model Registry Lifecycle States

  • Draft: registered from training run, not yet evaluated
  • Staging: passed automated tests, in manual review
  • Production: approved and serving live traffic
  • Archived: superseded by newer version, retained for audit

Automated Model Testing

Model testing gates have three layers. Data validation checks that the training data distribution has not shifted unexpectedly before training begins — feature distributions, missing value rates, class balance in classification tasks. A training run on corrupted or drifted data should fail before it wastes compute. Great Expectations and Evidently are well-maintained libraries for this layer.

Model quality testing runs the trained model against a fixed held-out evaluation set and compares metrics against defined thresholds. These thresholds should be set relative to the current production model, not absolute values: the new model must achieve at least 98% of the current model's accuracy on the primary metric, and must not regress more than 2% on any defined subgroup evaluation set. Absolute thresholds become stale as models improve; relative thresholds always compare to the current production baseline.

Regression testing maintains a curated set of labeled examples — typically 200 to 500 — covering edge cases, previously observed failure modes, and high-stakes scenarios. This set grows over time as production incidents are captured and added. Every new model must pass this regression suite before promotion. Unlike aggregate metrics, regression suites catch cases where a model improves on average but fails on specific known-important inputs.

Deployment Strategies

Shadow deployment runs the new model in parallel with the current production model, receives production traffic, and logs outputs without serving them to users. This is the safest validation strategy: you get real distribution data at production scale without any user-facing risk. The cost is doubled inference compute for the shadow period. Two to seven days of shadow deployment before promotion is a reasonable default for models making consequential decisions.

Canary deployment routes a small fraction of production traffic — typically 1–5% — to the new model and monitors output quality metrics, latency, and error rates before increasing the fraction. Unlike shadow deployment, canary results reflect actual user behaviour with the new model. The risk is that the initial canary population may be exposed to degraded outputs, so canary fractions should start small and increase based on monitoring confidence.

Blue-green deployment maintains two full production serving stacks and switches traffic atomically. It is the right choice for models where even a brief period of mixed outputs from two model versions is unacceptable — recommendation systems where consistency matters, or classification systems whose outputs feed downstream logic that assumes a single model's behaviour. Blue-green requires double the serving infrastructure during the promotion window.

Post-Deployment Monitoring

Production model monitoring has two components that software monitoring does not: output distribution monitoring and ground truth collection. Output distribution monitoring tracks whether the model's output distribution is shifting over time — a classifier whose confidence distribution changes without any code changes is receiving different inputs, which may indicate data drift requiring retraining. Evidently, WhyLogs, and Arize are established platforms for this layer.

Ground truth collection connects model outputs to eventual outcomes, enabling retrospective evaluation of production accuracy. For high-stakes applications, this is mandatory: a fraud detection model whose training accuracy is 95% but whose production accuracy is 73% (because fraud patterns have evolved) needs ground truth feedback to detect the divergence. The monitoring pipeline should compute rolling accuracy metrics over recent production data as ground truth labels become available.

Automated retraining triggers close the loop. When drift metrics exceed thresholds, or when rolling production accuracy falls below threshold, the pipeline should trigger an automatic retraining run, validate the result, and either auto-promote (if all tests pass and the improvement is unambiguous) or escalate for human review. The goal is to reduce the time between detecting degradation and restoring performance — from weeks in a manual workflow to hours in a well-automated one.