Read the overview of the approach here.

Batch monitoring relies on the core evaluation API of the Evidently Python library. Check the detailed guide.

​
Simple Example

You must first connect to Evidently Cloud and create a Project.

To get the dataset stats for a single batch and upload to the workspace:

eval_data = Dataset.from_pandas(
    pd.DataFrame(source_df),
    data_definition=DataDefinition()
)

report = Report([
    DatasetStats()
])

my_eval = report.run(eval_data, None)
ws.add_run(project.id, my_eval, include_data=False)

​
Workflow

The complete workflow looks as the following.

1

Configure the metrics

Define an Evidently Report with optional Test conditions to define the evals.

2

Run the evals

You must independently execute Reports on a chosen cadence. Consider tools like Airflow. You can send Reports from different steps in your pipeline. For example:

  • first, send data quality, data drift and prediction drift checks

  • after you get the delayed labels, send a ML quality checks results.

You can backdate your Reports with a custom timestamp.

3

Upload to the platform

Choose to store raw inferences or only upload the metric summaries. How to upload / delete results.

4

Configure the Dashboard

Set up a Dashboard to track results over time: using pre-built Tabs or configure your own choice of monitoring Panels. Check the Dashboard guide.

5

Configure alerts

Set up alerts on Metric values or Test failures. Check the section on Alerts.

Running Tests vs Reports. Structuring your evaluations as Tests - as opposed to monitoring lots of metrics at once - can help reduce alert fatigue and simplify configuration when evaluating multiple conditions at once. For example, you can quickly verify that all columns in the input data are within a defined min-max range.