Practice: Model evaluation: regression suites and release gates

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Main tool: MLflow · MLflow is the one tool here that keeps the evaluation dataset, the metrics, the baseline comparison, the threshold verdict and the registered model version in the same store, so a release gate can be recorded and audited where promotion happens. · Alternatives: Evidently generates drift and quality test reports but has no model registry; Great Expectations validates data assumptions and not model quality; CheckList generates behavioral test cases for NLP models only; plain pytest with scikit-learn metrics gives full control but keeps no run history or verdict attached to a model version; Locust load-tests a serving endpoint and nothing else.

Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1

Setup (one time)

1. Create a fresh Python virtual environment and install MLflow 3 together with scikit-learn and pandas.
The evaluation and validation APIs changed in MLflow 3, so the whole path assumes that major version. A clean environment keeps the gate reproducible later.
Explained in: MLflow 3 Migration Guide | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: Asking the mlflow command for its version prints a 3.x number.
2. Start a local MLflow server with the single mlflow command and open its UI in a browser.
Every later step records runs, datasets and verdicts to this server, and the UI is where you inspect them.
Explained in: Self Hosting Overview | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The MLflow UI loads on localhost and shows the Default experiment.
3. Point your Python session at the local server by setting the tracking URI, and create an experiment named for this lesson.
Without an explicit tracking URI the runs land in a local folder the server does not read. One named experiment keeps candidate and baseline runs side by side.
Explained in: ML Experiment Tracking | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The new experiment name appears in the UI's experiment list.

Steps

1. Run the model training quickstart to train and log a small scikit-learn classifier, then train a second, deliberately weaker one to act as the current production baseline.
A gate needs two things to compare: a candidate and the model it would replace. This first run also confirms that logging to your server works end to end.
Explained in: Getting Started with MLflow for ML | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: Two runs, each with a logged model, are visible in your experiment in the UI.
2. Run the mlflow evaluate API on the candidate model against a held-out dataframe, with the model type set to classifier.
This is the core object of the lesson: one call that takes a model, a dataset and a model type and returns a result object with metrics and artifacts.
Explained in: ML Model Evaluation | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The run shows built-in classifier metrics and plots such as a confusion matrix among its artifacts.
3. Write a custom metric that encodes the real decision, for example a cost-weighted error where a false negative costs more than a false positive, and pass it to the evaluation as an extra metric.
Default accuracy rarely matches what the business decides on. A gate is only useful when the gated number is the one that drives the promote or block decision.
Explained in: ML Model Evaluation | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: Your custom metric appears by name next to the built-in metrics on the evaluation run.
4. Create a frozen evaluation dataset from a fixed file, wrap it as an MLflow dataset with a name and a source, and run the evaluation on that dataset object in place of the raw dataframe.
A regression suite only means something when the data does not move between releases. Tracking the dataset gives every result a digest that proves which data was scored.
Explained in: ML Dataset Tracking | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The run's dataset panel in the UI shows the dataset name and digest, and a second evaluation shows the same digest.
5. Run a dataset evaluation on precomputed predictions once per slice, such as one run per customer segment or class, with the slice name recorded as a run tag.
An aggregate score can improve while one segment gets worse. Evaluating stored predictions per slice makes those regressions visible without reloading the model.
Explained in: ML Model Evaluation | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The experiment holds one evaluation run per slice, each with its own metric values and slice tag.
6. Query the runs programmatically to pull the candidate and baseline metrics for every slice into one table, and compare them side by side.
The gate must read persisted results, not numbers from a notebook cell. This query is what an automated check will later run.
Explained in: ML Experiment Tracking | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: You have a table listing candidate and baseline values of your decision metric for each slice, built only from the tracking server.
7. Write a behavioral test with function evaluation: wrap the model in a Python function that perturbs inputs in a way that should not change the answer, and evaluate it with a custom metric counting changed predictions.
Held-out accuracy does not tell you whether the model is stable under harmless input changes. Function evaluation lets you score any callable, so invariance and directional tests fit the same tracking flow.
Explained in: ML Model Evaluation | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: A run records an invariance failure rate metric, and it rises when you make the perturbation harsher.
8. Run model validation on the candidate's evaluation result with metric thresholds: an absolute floor on the decision metric and a minimum improvement over the baseline's result.
This turns good enough into a pass or fail that code can act on. Comparing against the baseline is what makes the suite a regression check and not only a quality bar.
Explained in: ML Model Evaluation | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: Validation passes for the candidate and raises a validation failure when you swap in the weaker model.
9. Register the candidate as a new version of a named model with the Python API, and attach the gate verdict, the dataset digest and the evaluation run id to that version as tags.
The verdict has to travel with the model version, so anyone promoting it can see what it passed and on which data. Only versions with a passing verdict should receive the alias that deployment reads.
Explained in: ML Model Registry | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The model version page in the UI shows the verdict tags and a link back to its source run.
10. Serve the registered version locally as a REST endpoint with MLflow's built-in model server, send it a fixed batch of requests, and log the measured p95 latency as a metric checked against a budget.
A model that scores well but answers too slowly still fails in production. Gating the served system catches packaging and latency problems that offline metrics cannot.
Explained in: MLflow AI Engineering Platform | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The local endpoint returns predictions, and a p95 latency metric is recorded with a pass or fail against your budget.
11. Write a pytest file that runs the frozen-dataset evaluation, the slice checks, the behavioral test and the threshold validation, then run it from a clean checkout with pinned package versions; the cited note describes the pytest integration for GenAI regression tests, and the same pattern applies here.
A gate that depends on someone remembering to run a notebook is not a gate. A test file with pinned versions runs identically on a laptop and on any CI runner, with no paid cloud needed.
Explained in: MLflow (mlflow.org), checked 2026-10-11
Done when: One pytest run exits green for the candidate and red for the weaker model, and each run leaves evaluation records on the server.
12. Deploy a production-like tracking setup locally, with the server backed by a database backend store and a separate artifact store in containers, then rerun the pytest gate against it.
Teams share one remote server, so the gate must work when metadata and artifacts live in separate stores behind a server. Containers give you that layout on a laptop.
Explained in: ML Experiment Tracking | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: After restarting the containers, the gate's runs, dataset records and tagged model version are all still visible in the UI.

Back to Model evaluation: regression suites and release gates