How to use Metric Presets in Evidently.
TL;DR: Evidently has pre-built Reports that work out of the box. To use them, simply pass your data and choose the Preset.
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset, DataQualityPreset
You need two datasets for comparison: reference and current. You can also generate some of the Reports with a single dataset.
Evidently has ready-made Metric Presets that group relevant Metrics in a single Report. You can use them as templates to evaluate a specific aspect of the data or model performance.
To use the Preset, create a
Reportobject and specify the chosen
presetin a list of
metrics. You should also point to the current and reference dataset (if available). If nothing else is specified, the Report will run with the default parameters for all columns in the dataset.
Example 1. To generate the Report that includes two Presets for Data and Target Drift:
drift_report = Report(metrics=[DataDriftPreset(), TargetDriftPreset()])
It will display the HTML combined report.
Example 2. To generate the Report for a single dataset:
data_quality_report = Report(metrics=[
data_quality_report.run(current_data=current, reference_data = None, column_mapping=None)
Aggregated visuals in plots. Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an option to generate plots with raw data. You can choose whether you want it on not based on the size of your dataset.
Here is a list of Metric Presets you can try:
You can get the Report results in different formats.
HTML. You can get an interactive visual report. It is best for exploration and debugging. You can also document the results and share them with the team.
To see in Jupyter notebook or Colab, call the object:
To export HTML as a separate file:
Reports contain interactive visualizations inside the HTML, so large reports might take time to load. In this case, downsample your data.
JSON. You can get the results of the calculation as a JSON. It is best for automation and integration in your prediction pipelines.
To get the JSON:
To export JSON as a separate file:
Python dictionary. You can get the output in the Python dictionary format. Using a Python object might be more convenient if you want to apply multiple transformations to the output.
To get the dictionary:
Evidently snapshot. You can also save the output as an Evidently JSON
snapshot. This will allow you to visualize the model or data quality over time using the Evidently ML monitoring dashboard.
You can customize some of the Presets using parameters. For example, you can calculate the quality metrics for a binary probabilistic classification model with a custom decision threshold:
dataset_report = Report(metrics=[