CPU hit 90% on a production ML host and the team found out from a customer ticket. Alerts existed — three overlapping alert systems, in fact — and they had all fired. Nobody saw them because the alert channel had been muted after too many false positives the previous month. This is the story of how we rebuilt anomaly detection for that system: from the alert archaeology that revealed what was broken, through the redesign that reduced signal noise by 80%, to the isolation forest that now pages the right engineer in under forty-five seconds for the patterns that matter.
The alert fatigue problem — and how it compounds
Three monitoring tools, overlapping thresholds, no designated owner. Each tool had been set up by a different team at a different stage of the project with good intentions and no coordination. By the time the CPU incident happened, the on-call channel received an average of 47 alert notifications per day, of which 3 required action. The other 44 were known patterns, known false positives, or duplicates of each other.
Alert fatigue is not laziness — it is a rational response to noise. When 93% of notifications require no action, the rational response is to deprioritise the channel. The problem is that the 7% that require action are distributed unpredictably. Muting the noise means muting the signal. The team was not negligent; the monitoring system had trained them to ignore it.
Static thresholds fail on ML workloads with natural variation. An alert on "CPU above 75%" fires every morning when the batch scoring job runs, every Monday when weekly reports trigger, and every time a deploy causes a brief spike during model warm-up. None of those are incidents. An alert that fires on those patterns will be ignored when the pattern is an actual anomaly — because the pattern looks identical to everything that was not an anomaly before it.
The archaeological work before rebuilding revealed that the team had attempted to fix this three times: raising thresholds to reduce false positives (which caused real incidents to slip through), adding more granular alerts (which increased volume), and creating a "critical only" channel (which was promptly overpopulated with alerts labelled critical to avoid the main channel). Each fix made the underlying problem worse.
Redesigning from first principles
We started by defining the incident taxonomy: what patterns should wake someone up at 2 a.m.? The answers were more specific than the existing alert system implied. Sustained CPU above 85% for more than five minutes, combined with queue depth growing. Inference error rate above 2% for more than two minutes. Any service returning 500s on health checks for more than thirty seconds. Everything else was interesting for investigation, not urgent for paging.
We deleted duplicate alerts before adding anything new. The single most impactful change in the first week was removing 31 of the 47 daily alerts that were redundant — covered by a surviving alert or describing a pattern that had never corresponded to an incident in the historical record. Starting from a clean baseline made everything else more effective.
Context was added to every surviving alert. Each notification linked directly to the relevant Grafana panel — not the dashboard homepage — pre-configured to show the 30-minute window around the alert time, with deploy markers, traffic shape, and queue depth visible simultaneously. On-call engineers should be able to form an initial hypothesis within two minutes of the notification without opening a second tool.
Runbooks were written for every alert that remained active. A runbook describes three things: what this alert means, what to check first, and the most common resolution paths. Runbooks linked from the alert notification and updated within 48 hours of any incident that revealed a gap. Stale runbooks are worse than no runbooks.
The isolation forest approach for dynamic baselines
Static thresholds handle hard limits well: disk fill rate, health check failures, error budget exhaustion. They handle dynamic ML workloads poorly. We added an isolation forest trained on two weeks of normal operational data to handle the patterns that static thresholds kept misfiring on.
The isolation forest treats CPU utilisation, queue depth, inference latency p95, and request rate as a joint distribution. Normal operation occupies a region in that joint space that reflects the actual patterns of the workload — including the Monday morning spike, the batch job at 3 a.m., and the post-deploy warm-up shape. Anomalies are points that fall far from that region, regardless of whether any individual metric crosses a static limit.
The model retrains weekly on a rolling window. This handles gradual drift — a service that legitimately grows in baseline traffic over months — without requiring manual threshold updates. The retraining runs automatically, the new model is compared against historical incidents before promotion, and rollback to the previous version is available for 30 days.
We instrument the anomaly score itself in Prometheus alongside the raw metrics. When a page fires, the Grafana panel shows the anomaly score time series overlaid on the component metrics — so on-call engineers can see both what the model flagged and which raw metric drove the score. Opaque ML-generated alerts undermine trust; transparent ones build it.
Results, calibration, and what ongoing maintenance looks like
Alert volume dropped from 47 per day to 8 per day. Actionable percentage increased from 7% to 75%. The CPU incident that prompted the rebuild would have been caught in under 45 seconds on the new system — the anomaly score exceeded the threshold 14 minutes before the customer ticket arrived.
False positive rate in the first month after deployment was 22% — higher than the target but lower than the pre-rebuild rate on a per-alert basis. We calibrated the alert sensitivity upward over weeks two and three as the team built confidence in the system and identified patterns that were being flagged incorrectly. Calibration is an ongoing process, not a launch-and-forget activity.
The on-call rotation reported higher confidence in the alert system after six weeks — measured not by survey but by response time. Average response to a page went from 9 minutes (team checking whether the alert was real before reacting) to under 2 minutes. When engineers trust the signal, they act on it.
The weekly retraining cycle caught a gradual baseline shift three months after deployment when a new feature caused a sustained 15% increase in average CPU utilisation. The old static threshold would have generated a false positive every day for that period. The updated baseline model absorbed the new normal within one retraining cycle with no manual intervention.
