Data drift algorithm
In some tests and metrics, Evidently uses the default Data Drift Detection algorithm. It helps detect the distribution drift in the individual features, prediction, or target.

How it works

Evidently compares the distributions of the values in each column of the two datasets. You should pass these datasets as reference and current. Evidently applies several statistical tests and metrics to detect if the distribution has changed significantly. It returns a "drift detected" or "not detected" result for each column.
There is a default logic to choosing the appropriate drift test for each column. It is based on:
  • column type: categorical or numerical
  • the number of observations in the reference dataset
  • the number of unique values in the column (n_unique)
For small data with <= 1000 observations in the reference dataset:
  • For numerical columns (n_unique > 5): two-sample Kolmogorov-Smirnov test.
  • For categorical columns or numerical columns with n_unique <= 5: chi-squared test.
  • For binary categorical features (n_unique <= 2): proportion difference test for independent samples based on Z-score.
All tests use a 0.95 confidence level by default.
For larger data with > 1000 observations in the reference dataset:
All metrics use a threshold = 0.1 by default.
You can always modify this drift detection logic. You can select any of the statistical tests available in the library (including PSI, K--L divergence, Jensen-Shannon distance, Wasserstein distance, etc.), specify custom thresholds, or pass a custom test.

Dataset-level drift

The method above calculates drift for each column individually.
To detect dataset-level drift, you can set a rule on top of the individual feature results. For example, you can declare dataset drift if at least 50% of all features drifted or if ⅓ of the most important features drifted. Some of the Evidently tests and presets include such defaults. You can always modify them and set custom parameters.

Resources

To build up a better intuition for which tests are better in different kinds of use cases, you can read our in-depth blog with experimental code:
Additional links:
Copy link
Outline
How it works
Dataset-level drift
Resources