Embeddings drift parameters
How to customize data drift detection for embeddings.
Pre-requisites:
You know how to generate Reports or Test Suites with default parameters.
You know how to pass custom parameters for Reports or Test Suites.
You know how to use Column Mapping to map embeddings in the input data.
Code example
You can refer to an example How-to-notebook showing how to pass parameters for different embeddings drift detection methods:
Default
When you calculate embeddings drift, Evidently automatically applies the default drift detection method (“model”).
In Reports:
In Test Suites:
It works the same inside presets, like DataDriftPreset
.
Embedding parameters - Metrics and Tests
You can override the defaults by passing a custom drift_method
parameter to the relevant Metric or Test. You can define the embeddings drift detection method, the threshold, or both.
Pass the drift_method
parameter:
Embedding parameters - Presets
When you use NoTargetPerformanceTestPreset
, DataDriftTestPreset
or DataDriftPreset
you can specify which subsets of columns with embeddings to include using embeddings
, and the drift detection method using embeddings_drift_method
.
By default, the Presets will include all columns mapped as containing embeddings in column_mapping
.
To exclude columns with embeddings:
To specify which sets of columns to include (with the default drift detection method):
To specify which sets of columns to include, and specify the method:
Embedding drift detection methods
Currently 4 embeddings drift detection methods are available.
Embeddings drift detection method | Description and default |
---|---|
|
|
|
|
|
|
|
|
If you specify an embedding drift detection method but do not pass additional parameters, defaults will apply.
You can also specify parameters for any chosen method. Since the methods are different, each has a different set of parameters. Note that you should pass the parameters directly to the chosen drift detection method, not to the Metric.
Model-based (“model”)
Parameter | Description |
---|---|
| Sets the threshold for drift detection (ROC AUC). Drift is detected when |
| Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing. If applied, the ROC AUC of the classifier is compared to the ROC AUC of the random classifier at a set percentile. The calculation is repeated 1000 times with randomly assigned target class probabilities. This produces a distribution of random roc_auc scores with a mean of 0,5. We then take the 95th percentile (default) of this distribution and compare it to the ROC-AUC score of the classifier. If the classifier score is higher, data drift is detected. Default: True if <= 1000 objects, False if > 1000 objects. |
| Sets the percentile of the possible ROC AUC values of the random classifier to compare against. This applies when bootstrap is True. Default: 0.95 |
| The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of |
Maximum mean discrepancy (“mmd”)
Parameter | Description |
---|---|
| Sets the threshold value of MMD for drift detection. Drift is detected when |
| Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing. If applied, the value of MMD between reference and current (mmd_0) is tested against possible MMD values in reference. We randomly split the reference data into two parts and compute MMD values (mmd_i) between them. The calculation is repeated 100 times. This produces a distribution of MMD values obtained for a reference dataset. We then take the 95th percentile (default) of this distribution and compare it to the MMD between reference and current datasets. If the mmd_0 > mmd_95, data drift is detected. Default: True if <= 1000 objects, False if > 1000 objects. |
| Sets the percentile of the possible MMD values in reference to compare against.
Applies when |
| The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of |
Share of drifted embedding components (“ratio”)
Parameter | Description |
---|---|
| Sets the tabular drift detection method (any of the tabular drift detection methods for numerical features available in Evidently). Default: Wasserstein |
| Sets the threshold for drift detection for individual embedding components. Drift is detected when |
| Sets the threshold (share of drifted embedding components) for drift detection for the overall dataset. Default: 0.2 |
| The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of |
Distance-based methods (“distance”)
Parameter | Description |
---|---|
| Sets the distance metric for drift detection. Default: Euclidean distance |
| Sets the threshold for drift detection. Drift is detected when |
| Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing. If applied, the distance between reference and current is tested against possible distance values in reference. We randomly split the reference data into two parts and compute the distance between them. The calculation is repeated 100 times. This produces a distribution of distance values obtained for a reference dataset. We then take the 95th percentile (default) of this distribution and compare it to the distance between reference and current datasets. If the distance between the reference and current is higher than the 95th percentile of the distance obtained for the reference dataset, the drift is detected. Default: True if <= 1000 objects, False if > 1000 objects. |
| Sets the percentile of the possible distance values in reference to compare against.
Applies when |
| The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of |
Last updated