Quickstart for ML Monitoring
In this tutorial, you'll learn how to start with Evidently ML monitoring. You will launch a locally hosted dashboard to visualize the performance of a toy model.
The tutorial is split into two parts.
Part 1. Estimated time: 2 minutes. You will launch a demo example.
- Install Evidently
- Launch a pre-built demo project
Part 2. Estimated time: 15 minutes. You will add a new project to the monitoring UI.
- Create a new workspace and project
- Imitate batch model inference
- Design a few monitoring panels
- View the dashboards
Note: This guide assumes you run Evidently locally.
Optional, but highly recommended. Create a virtual environment and activate it. Run the following command in the Terminal:
pip install virtualenv
Evidently is available as a PyPi package. Run this command to install Evidently:
pip install evidently
You can also install Evidently from Conda:
conda install -c conda-forge evidently
Note: ML monitoring is available starting from Evidently version 0.4.0.
To launch the Evidently service with the demo project, run:
evidently ui --demo-project
To view the Evidently interface, go to URL http://localhost:8000 in your web browser.
You will see a pre-built project that shows the toy model quality over 20 days. You can switch between tabs and view individual Reports and Test Suites. Each Report or Test Suite contains the information logged for a daily period. The monitoring dashboard parses the data from these Reports and shows how metrics change over time.
What is a Test Suite and a Report? If you are new to Evidently, we recommend that you go through the Quickstart for Tests and Reports next. This functionality helps log data and model metrics for ML monitoring.
In this section, you will create a new project as if you add a new ML model to monitor. You will imitate batch model inference to collect logs and design panels to visualize them. All steps are available as a Python script.
You can access it here:
Note: we assume you already installed Evidently at the previous step.
Open and explore the
This script does the following:
- Imports the required components
- Imports a toy dataset
- Defines the metrics to log using Evidently Reports and Test Suites
- Computes the metrics iterating over toy data
- Creates a new Evidently workspace and project
- Creates several panels to visualize the metrics
You can simply run this script without changes. The sections below are optional. They describe each step in the script and what you can change – if you are up to it!
Import the required components.
from sklearn import datasets
from evidently.metrics import ColumnDriftMetric
from evidently.metrics import ColumnSummaryMetric
from evidently.metrics import DatasetDriftMetric
from evidently.metrics import DatasetMissingValuesMetric
from evidently.report import Report
from evidently.test_preset import DataDriftTestPreset
from evidently.test_suite import TestSuite
from evidently.ui.dashboards import CounterAgg
from evidently.ui.dashboards import DashboardPanelCounter
from evidently.ui.dashboards import DashboardPanelPlot
from evidently.ui.dashboards import PanelValue
from evidently.ui.dashboards import PlotType
from evidently.ui.dashboards import ReportFilter
from evidently.ui.remote import RemoteWorkspace
from evidently.ui.workspace import Workspace
from evidently.ui.workspace import WorkspaceBase
Import the data and create a
pandas.DataFrameusing the OpenML
We single out the part of the dataset
adult_refthat we will later use as a baseline for drift detection. We use the rest
adult_curto imitate batch inference.
adult_data = datasets.fetch_openml(name="adult", version=2, as_frame="auto")
adult = adult_data.frame
adult_ref = adult[~adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
adult_cur = adult[adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
This step is added for demo purposes. In practice, you should work with production data.
Assign a name to the Evidently workspace and your project. A project corresponds to an ML model you monitor. You will see this name and description in the user interface.
WORKSPACE = "workspace"
YOUR_PROJECT_NAME = "New Project"
YOUR_PROJECT_DESCRIPTION = "Test project using Adult dataset."
Workspace defines the folder where Evidently will log data to. It will be created in the directory where you launch the script from.
To capture data and model metrics, we use Evidently Reports and Test Suites. You need to define a Report and/or a Test Suite object and list the metrics or tests to include. You can pass optional parameters – such as set a drift detection method instead of the default choice.
Test Suites and Reports. We use the same syntax to log the data as when running Evidently Reports and Tests Suites ad hoc, for example, in a Jupyter notebook. Go through the Quickstart if you need a refresher.
To imitate batch inference, we run computations as if we captured data for
idays (we will set it to 5), each time taking 100 observations. In practice, you must work with actual prediction data.
We also pass
reference_datato the Evidently Report. It will be used as the basis of comparison for distribution drift detection.
def create_report(i: int):
data_drift_report = Report(
timestamp=datetime.datetime.now() + datetime.timedelta(days=i),
data_drift_report.run(reference_data=adult_ref, current_data=adult_cur.iloc[100 * i : 100 * (i + 1), :])
What you can change:
- The complete script uses both Test Suite and Report for logging. It is not required to use both – this depends on your preference.
- You can pass additional parameters to the individual Tests and Metrics.
Evaluating model quality. For simplicity, the example works with a raw dataset. There is no model! You cannot compute model quality metrics on this data – you need to add prediction and target columns. You might also need to use column mapping to map your inputs.
Define how you create a new project in the workspace:
def create_project(workspace: WorkspaceBase):
project = workspace.create_project(YOUR_PROJECT_NAME)
project.description = YOUR_PROJECT_DESCRIPTION
Each project can include multiple panels that appear on the monitoring dashboard. A panel visualizes a particular metric or metrics captured in Reports or Test Suites over time. You can visualize metrics as counters, time series plots, bar charts, and scatter plots.
Here is an example of adding a counter metric. The complete script includes several more.
title="Share of Drifted Features",
metric_id, you pass the name of the Evidently Metric that was logged as part of the Report. As a
field_path, you select the metric result computed as part of this Metric. Since Evidently Metrics contain multiple data points, you must specify which to show on a dashboard. You can display multiple metrics on a single panel.
What you can change:
- You can specify the panel title and the legend visible in the interface.
- You can set the size of each panel as full-width (
size=2, default) or half-width (
- You can define the aggregation function or select to show the last metric value.
- You can add panels of different types following the examples in the script.
To save changes made to a project, you must use the method save().
Finally, create the project, workspace, and generate the JSON
Snapshotis a JSON "version" of the
Test Suitethat was defined earlier. It contains summaries of the captured data and model metrics. You must store the snapshots in a directory under the corresponding workspace name. This way, Evidently UI will be able to parse the metrics and visualize them on the monitoring panels.
When you execute the script, Evidently will log the snapshots with the selected metrics to the defined workspace folder, as if you captured data for 5 days. It will also create the dashboard panels.
def create_demo_project(workspace: str):
ws = Workspace.create(workspace)
project = create_project(ws)
for i in range(0, 5):
report = create_report(i=i)
test_suite = create_test_suite(i=i)
if __name__ == "__main__":
If you made any edits to this script, save them.
If this is not the first run of the script, and you reuse the same project – run the command to delete a previously generated workspace:
rm -r workspace
Run the command to generate a new example project as defined in the script above.
Finally, launch the user interface that will include the defined project.
4.1. If you only want to include your project, run:
4.2. If you want to see both your new project and a standard demo project, run:
evidently ui –-demo-project
Note: If you already launched a demo project previously, it will remain in the workspace. There is no need to add it the second time.
4.3. If you want to have your project in the specified workspace and have the UI service running at the specific port:
evidently ui --workspace ./workspace --port 8080
4.4. If you want to see both your project and demo project in the specified workspace and run the UI service at the specific port:
evidently ui --workspace ./workspace --port 8080 –-demo-project
Note: Evidently collects anonymous telemetry about service usage. You can opt-out as described here. We’d greatly appreciate it if you keep the telemetry on since it allows us to understand usage patterns and continue building useful free tools for the community.
Access Evidently UI service in your browser to see the dashboards for the new project. Go to the
localhost:8000, or a specified port if you set a different one.
- Go through the steps in more detail
- Build the workflow
If you want to enable monitoring for an existing ML project, you must collect the data from your production pipelines or services, or run monitoring jobs over production logs stored in a data warehouse. The exact integration scenario depends on the model deployment type and infrastructure.
Here is a possible approach for batch monitoring. You can implement it using a workflow manager like Airflow to compute Evidently snapshots on a regular cadence.