Practice: Model monitoring: data quality, drift and alerting
Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.
Main tool: Amazon SageMaker · Amazon SageMaker documents the full monitoring loop in one place (capture, baseline, scheduled checks, violations, metrics and alarms), and its availability-change page maps each part to open-source replacements you can run on a laptop. · Alternatives: Evidently is an open-source library that computes quality and drift reports anywhere Python runs; whylogs logs compact statistical profiles for later comparison; Great Expectations validates data against declared expectations but does not measure drift statistically; Prometheus with Alertmanager and Grafana handles metric storage, alert routing and dashboards but needs you to compute the model metrics yourself; Amazon CloudWatch is the managed metric and alarm service that SageMaker monitoring emits into.
Every step links into the tool's own documentation; links were checked on 2026-10-12 · version 1
Setup (one time)
1. Read the Model Monitor availability change notice and note whether your account can still create new monitoring schedules.
The notice decides which parts of the path you practise on the managed service and which on the documented open-source replacements. Knowing it first avoids building on a feature you cannot enable.
Explained in: Amazon SageMaker Model Monitor availability change - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a written note stating which monitor types you will practise locally and which, if any, in a cloud account.
2. Read how Model Monitor works and list the artifacts it produces: captured data, baseline statistics, constraints, violations and metrics.
Every later step creates or reads one of these artifacts. Having the names fixed makes the local substitutes easy to map back to the managed service.
Explained in: Data and model quality monitoring with Amazon SageMaker Model Monitor - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a list of five artifacts with one line on who writes each and who reads it.
3. Choose the monitoring solution you will run on the laptop, using the comparison of the open-source and CloudWatch options.
SageMaker is a paid cloud service, so the laptop path uses the documented replacement closest to it. Choosing once keeps the remaining steps consistent.
Explained in: Amazon SageMaker Model Monitor availability change - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: You can state your chosen option and one reason it fits a laptop with no paid cloud.
4. Install a pinned Python environment with the SageMaker Python SDK and the open-source monitoring library named in the getting started section, plus a small tabular dataset split into a training part and a serving part.
The same environment runs every step and is what you pin later for automation. The split gives you a baseline dataset and a current dataset.
Explained in: Amazon SageMaker Model Monitor availability change - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: Importing both packages succeeds in a fresh shell and two dataset files exist on disk.
Steps
1. Run a first data quality report locally, comparing the serving part of your dataset with the training part, following the open-source option for data quality.
This is the shortest route to seeing what a monitoring result looks like. It needs no cloud account and no GPU.
Explained in: Amazon SageMaker Model Monitor availability change - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: A report file opens in your browser showing per-column statistics for both datasets.
2. Write a one-page map of the objects a private monitoring setup connects (service provider, VPC endpoint, endpoint network interface, endpoint policy), marking where the inference endpoint, the capture bucket and the monitoring job sit.
Monitoring jobs in a locked-down account reach the model endpoint and storage through private endpoints. An ML platform engineer has to know these objects before a schedule can run at all.
Explained in: AWS PrivateLink concepts - Amazon Virtual Private Cloud (docs.aws.amazon.com), checked 2026-10-12
Done when: Your map names each object and shows which side of the connection the monitoring job is on.
3. Create a baseline from the training dataset: per-feature statistics and a set of suggested constraints. The managed baseline job needs a paid cloud account, so on the laptop generate the same two files with the open-source library and keep them under version control.
Data quality monitoring means nothing without a baseline to compare against. The constraints are the contract every later batch is checked against.
Explained in: Create a Baseline - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: Two files exist, one with statistics and one with constraints, and you can point at the constraint for one feature.
4. Compare a corrupted serving batch (missing values, a changed type, an out-of-range column) against the baseline and write the result in the documented violations layout.
This is data quality monitoring against a baseline in its simplest form. Using the documented violation types makes your local output read like the managed one.
Explained in: Schema for Violations (constraint_violations.json file) - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: Your violations file lists at least three violations, each with a feature name, a check type and a description.
5. Create a shifted serving batch and run drift checks on it, then label each finding with the monitor type it belongs to: data quality drift, model quality drift, bias drift or feature attribution drift.
The kinds of drift are measured differently and call for different responses. Sorting real findings into the four types fixes the vocabulary.
Explained in: Data and model quality monitoring with Amazon SageMaker Model Monitor - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a table with one row per finding, the distance or metric used and one of the four drift types.
6. Configure data capture for your model: a real endpoint needs a paid cloud account, so on the laptop wrap a local prediction function to append each request and response to a dated file with a unique inference id.
Monitoring can only analyse what was recorded. The inference id is what later makes it possible to attach ground truth to a prediction.
Explained in: Data capture - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: After twenty local predictions the capture file holds twenty records, each with input, output, timestamp and inference id.
7. Write delayed ground truth labels keyed by inference id, join them to the captured predictions and compute model quality metrics against a baseline of the same metrics.
Accuracy can only be monitored once labels arrive, often days later. The join is where most model quality pipelines break.
Explained in: Model quality - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: A merged table shows prediction and label side by side, and a metric such as accuracy is printed for the baseline and the current window.
8. Configure an alerting rule on one drift metric and one quality metric with a threshold and an evaluation period. CloudWatch alarms need a cloud account, so on the laptop emit the metrics to a local file and have a script exit with a failure when the threshold is breached.
A dashboard nobody watches does not protect a model. The rule turns a metric into a page or a retraining trigger.
Explained in: CloudWatch Metrics - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: The clean batch passes the rule and the shifted batch makes the script fail with a message naming the metric and threshold.
9. Query the results of your last runs the way the documentation lists executions and opens the violations report, then write a short response runbook: who is paged, what is checked first, and when retraining is triggered.
Alerting is only half the job; the response has to be decided before the alert fires. Reading the violations report is the first action in any such runbook.
Explained in: Interpret results - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: Your runbook maps each violation type from your report to a named first action and an escalation rule.
10. Reproduce the whole check as a scheduled job: baseline files from version control, the pinned environment, the latest capture window in, a report and an exit status out. A managed monitoring schedule needs a cloud account, so use a local scheduler or a continuous integration job on a fixed interval.
Monitoring that runs only when someone remembers is not monitoring. Scheduling on a fixed window is also what makes results comparable between runs.
Explained in: Schedule monitoring jobs - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: Two consecutive scheduled runs produce two dated reports with no manual action.
11. Run a segment-sliced bias check and a feature attribution comparison on the captured data. The managed bias and explainability monitors need a paid cloud account, so compute per-segment metrics and feature importance rankings locally against the baseline.
A model can keep its overall accuracy while degrading for one group or relying on different features. These two monitors catch what data and quality checks miss.
Explained in: Bias drift for models in production - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: You have one per-segment metric table and one ranked feature importance list, each for baseline and current data, with differences flagged.
12. Deploy the production-like setup end to end on the laptop: a containerised model service with capture on, the scheduled monitoring job, the ground truth join, metrics emitted, the alert rule and the runbook, then feed it drifting traffic for several windows.
Running all the parts together exposes the gaps between them, such as late labels and empty windows. This is the shape described for monitoring a model in production.
Explained in: Monitoring a Model in Production - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-12
Done when: A drifting traffic window produces a violations report and a fired alert without manual steps, and a clean window produces neither.