Most AI and analytics projects stall not on model architecture but on data plumbing. The pipeline that was supposed to deliver clean training data is running a 50-year-old cron job on a server nobody knows the password for. The integration that feeds the dashboard was built by someone who left the company and is held together by an undocumented JSON transformation in a shared Google Sheet. These are not exotic problems — they are the normal state of enterprise data infrastructure after ten years of accumulation. This article covers the engineering approach we use to unblock those situations without a rip-and-replace fantasy that collapses under its own scope.
ETL pipeline design that operations teams can trust
Idempotency is the most important property of a production data pipeline. An idempotent pipeline produces the same output when re-run on the same input, regardless of how many times it has previously run. This means a failed pipeline run can be safely retried without producing duplicate records, phantom deletes, or inconsistent state. Without idempotency, pipeline failures become operational crises requiring manual intervention to assess and repair damage.
Ownership is explicit and documented for every pipeline stage. Who owns the schema of the source data? Who is responsible for the transformation logic? Who is notified when the pipeline fails? Pipelines without clear ownership accumulate silent failures and drift that nobody notices until downstream consumers report wrong numbers. "Owned by the data team" is not an ownership model — a named individual with an on-call rotation is.
Schema changes in source systems are treated as breaking changes with a migration path, not as events the pipeline should silently accommodate. A source column that is renamed without a migration plan silently nullifies every downstream metric that references it. We implement schema version contracts at pipeline ingestion points and alert on contract violations rather than consuming malformed data and propagating the corruption downstream.
Backfill capability is a requirement, not a feature. When a pipeline bug is discovered that produced incorrect output for the past three weeks, the question is not just "how do we fix the pipeline?" but "how do we correct the three weeks of wrong data?" A pipeline design that supports parameterised backfill — reprocessing a date range with the corrected logic — makes the answer tractable. A pipeline without backfill support makes it a multi-day project.
Monitoring data pipelines for freshness, not just uptime
A pipeline that runs successfully but produces output 18 hours late is not a healthy pipeline. Uptime monitoring tells you whether the pipeline process is running; freshness monitoring tells you whether the output data reflects recent source state. The difference matters for any downstream consumer that makes time-sensitive decisions from the data — and most analytics and ML consumers do.
Freshness SLOs are defined for each dataset: "the hourly sales table should reflect data no more than 90 minutes old by the top of each hour." These SLOs drive alerts that page the data team when the freshness target is missed — not when the pipeline process fails, which may not correlate. A pipeline that completes successfully on stale source data produces fresh-looking output that is actually stale.
Row count and distribution checks catch silent data quality regressions that completeness monitoring does not. A pipeline that produces the expected row count but with a dramatically different distribution of values in a key column has experienced a data quality event — a source change, a transformation bug, or a filter that changed behaviour. Great Expectations or dbt tests applied at each pipeline stage surface these regressions before they reach dashboards and ML model inputs.
Legacy modernization in slices — not rip-and-replace
Rip-and-replace is compelling on a whiteboard and catastrophic in practice. The plan is to rewrite the entire legacy system in parallel with the old one running, switch over when the new one is feature-complete, and retire the old one immediately. The reality is that feature-completeness with a legacy system of any complexity takes longer than estimated, the parallel operation period extends indefinitely, and the organisation ends up running two systems permanently at double the operational cost.
The strangler fig pattern delivers value incrementally while retiring risk progressively. A thin routing layer sits in front of the legacy system; new implementations of individual features or domains are built behind that layer and traffic is shifted to them as they become available. The legacy system is never fully rewritten — it is progressively deprioritised until the portions that remain are small enough to decommission without drama.
Slice selection prioritises high-risk legacy components, not low-risk ones. Modernising the low-risk, low-traffic parts of a legacy system first produces technical progress that creates no business value and retires no risk. The right first slice is the component that causes the most operational pain, creates the most security risk, or blocks the most important new capability.
Data migration from legacy to modern systems runs in parallel with the new implementation, not as a final step before cutover. Discovering data quality issues in the legacy system during migration — duplicate records, null values in required fields, character encoding problems — during the implementation phase allows them to be addressed with engineering time. Discovering them the night before cutover is an incident.
Third-party integrations that survive vendor changes
Every third-party integration is wrapped in an adapter layer that exposes a domain model rather than the vendor API directly. Application code that depends on the domain model is isolated from vendor API changes; when the vendor updates their API, the adapter changes and the application code does not. Without this isolation, a vendor API change affects every point in the codebase that calls it directly.
Integration tests against third-party services run against sandbox environments with recorded responses, not against production APIs. CI pipelines that call production APIs for test coverage create test rate limits, billing events, and flakiness from external network conditions. Recorded responses are deterministic, free, and fast — with the trade-off that they need to be updated when the vendor changes their response shape.
Vendor API contracts are documented and monitored for changes. Undocumented API changes are the most common cause of integration failures in production. We subscribe to vendor changelogs, monitor response shape against expected schemas, and alert on schema violations so API changes are detected before they cause downstream failures in user-facing features.
