The model had been "deployed" for three weeks before anyone noticed latency creeping up on one region. Not because the team was careless — because metrics lived in three separate tools, no one owned the dashboards, and the alerts had been muted after too many false positives. We rebuilt observability as one coherent story: the right metrics in Prometheus, dashboards in Grafana that the team actually opens, and ELK for the sessions that need end-to-end replay. This is what that looks like in practice for an ML service handling production traffic.
What to measure — and what to skip
Inference p95 and p99 latency, error rate by model version, queue depth, and GPU or CPU saturation are table stakes for any ML service. If you cannot see those four numbers in under thirty seconds, you are operating blind. Everything else is secondary until those are solid and stable.
For RAG and retrieval-augmented services, retrieval hit rate and empty-context response rate are as important as generation latency. A service can show perfect inference metrics while silently serving answers from empty or stale context — the model will generate something plausible either way. Retrieval metrics are where document rot and index drift appear first.
Data drift indicators do not need to be sophisticated on day one. Tracking weekly distributions on input token length, query category, and confidence score and alerting on step changes you cannot explain catches most silent regressions early. A sudden shift in the proportion of low-confidence answers often means a document changed — not that the model degraded.
We explicitly chose not to instrument every possible metric at launch. Too many metrics means too many dashboards nobody reads and too many alerts nobody trusts. We added instrumentation when a real incident revealed a gap — not in anticipation of incidents that had not happened yet.
How we wired Prometheus, Grafana, and ELK
Prometheus scrapes the ML service endpoints and infrastructure nodes on a 15-second interval. Custom metrics are exposed via a lightweight middleware layer so the service code itself does not import Prometheus libraries directly — decoupling that matters when the service runs in multiple environments with different collection configurations.
Grafana holds the dashboards. We use three tiers: an operational overview for on-call engineers (five panels, no scrolling required), a model performance board for the ML team (latency, quality metrics, version comparison), and a deep-dive board for incidents (trace links, error breakdown, deploy markers). Each tier has a different audience and a different refresh interval.
ELK catches what metrics miss. We log every inference request with a correlation ID, the retrieved chunks (hashed for privacy), the generation parameters, and the final response category. When a bad session gets flagged — by a user, by the weekly eval, or by an anomaly alert — we can replay the exact retrieval path that produced it. That traceability turned incident root-cause from a two-day exercise into a twenty-minute one.
Retention is set deliberately: Prometheus metrics for 90 days, Grafana snapshots for 180, ELK inference logs for 30 days with cold storage for 12 months. Long-lived metric retention matters for capacity planning and quarterly reviews; long-lived log retention matters for compliance audit trails. Treating both as the same problem leads to either cost waste or compliance gaps.
Alert design that teams actually follow
We deleted duplicate alerts instead of adding louder ones. Before the rebuild, the team had overlapping alerts from three tools covering the same signals with different thresholds. The first action was consolidation: one alert per signal, one on-call channel, severity mapped to a runbook. Reducing alert volume by 60% increased the team's response rate to remaining alerts by a measurable margin within a week.
Every alert links directly to a runbook. Not to the monitoring tool homepage — to the specific runbook for that alert pattern, with the most recent update date visible. If a runbook has not been updated in six months, the alert is reviewed before the next on-call rotation. Runbooks that go stale become the reason on-call engineers ignore pages.
Static thresholds fail on ML workloads with natural traffic patterns — morning surges, batch windows, weekly cycles. We use rolling baselines for latency and throughput alerts, reserving static thresholds for hard limits like disk fill rate and error budget exhaustion. A p95 alert that fires every Monday morning because traffic spikes on Monday is worse than no alert at all.
The test for a healthy alert system is simple: when a page fires at 2 a.m., does the on-call engineer know what to do within two minutes of reading the notification? If the answer requires opening three dashboards and reading a Slack thread, the alert is not finished — it is a starting point that has not been engineered yet.
