All tests

List of all Tests and Test Presets available in Evidently.

How to use this page

This is a reference page. You can return here:

• To discover available Tests and choose which to include in a custom Test suite.

• To understand which parameters you can change for a specific Test or Preset.

• To verify which tests are included in a Test Preset.

You can use the menu on the right to navigate the sections.

How to read the tables

• Name: the name of the Test or Test preset.

• Description: plain text explanation. For Tests, we specify whether it applies to the whole dataset or individual columns.

• Parameters: available configurations.

  • Required parameters are necessary for calculations, e.g. a column name for a column-level test.

  • Optional parameters modify how the underlying metric is calculated, e.g. which statistical test or correlation method is used.

  • Test condition parameters set the conditions (e.g. equal, not equal, greater than, etc.) that define the expectations from the Test output. If the condition is violated, the test returns a fail. They apply to most of the Tests, and are optional.

• Default Test condition: they apply if you do not set a custom сondition.

  • With reference: the Test conditions that apply when you pass a reference dataset and Evidently can derive conditions from it. (E.g. expect +/- 10% from reference).

  • No reference: the Test conditions that apply if you do not provide the reference. They are based on heuristics.

Test visualizations. Each Test also includes a default render. If you want to see the visualization, navigate to the example notebooks.

We are doing our best to maintain this page up to date. In case of discrepancies, consult the API reference or the "All tests" notebook in the Examples section. If you notice an error, please send us a pull request to update the documentation!

Test Presets

Default conditions for each Test in the Preset match the Test's defaults. You can see them in the tables below. The listed Preset parameters apply to the relevant individual Tests inside the Preset.

NoTargetPerformance Test Preset

Preset name: NoTargetPerformanceTestPreset()

Composition:

• TestShareOfDriftedColumns()

• TestColumnDrift(column_name=prediction)

• TestColumnShareOfMissingValues() for all or сolumns if provided

• TestShareOfOutRangeValues() for all numerical or specified columns

• TestShareOfOutListValues() for all categorical or specified columns

• TestMeanInNSigmas() for all numerical or specified columns

Optional parameters:

• columns

• stattest

• cat_stattest

• num_stattest

• per_column_stattest

• text_stattest

• stattest_threshold

• cat_stattest_threshold

• num_stattest_threshold

• per_column_stattest_threshold

• text_stattest_threshold

• embeddings

• embeddings_drift_method

• drift_share

How to set data drift parameters, embeddings drift parameters.

Data Stability Test Preset

Preset name: DataStabilityTestPreset()

Composition:

• TestNumberOfRows()

• TestNumberOfColumns()

• TestColumnsType()

• TestColumnShareOfMissingValues() for all or specified columns

• TestShareOfOutRangeValues() for all numerical or specified columns

• TestShareOfOutListValues() for all categorical or specified columns

• TestMeanInNSigmas() for all numerical or specified columns

Optional parameters:

• columns

Data Quality Test Preset

Preset name: DataQualityTestPreset()

Composition:

• TestColumnShareOfMissingValues() for all or specified columns

• TestMostCommonValueShare() for all or specified columns

• TestNumberOfConstantColumns()

• TestNumberOfDuplicatedColumns()

• TestNumberOfDuplicatedRows()

Optional parameters:

• columns

Data Drift Test Preset

Preset name: DataDriftTestPreset()

Composition:

• TestShareOfDriftedColumns()

• TestColumnDrift() for all or specified columns

Optional parameters:

• columns

• stattest

• cat_stattest

• num_stattest

• per_column_stattest

• text_stattest

• stattest_threshold

• cat_stattest_threshold

• num_stattest_threshold

• per_column_stattest_threshold

• text_stattest_threshold

• embeddings

• embeddings_drift_method

• drift_share

How to set data drift parameters, embeddings drift parameters.

Regression Test Preset

Preset name: RegressionTestPreset()

Composition:

• TestValueMeanError()

• TestValueMAE()

• TestValueRMSE()

• TestValueMAPE()

Optional parameters: N/A

Multiclass Classification Test Preset

Preset name: MulticlassClassificationTestPreset()

Composition:

• TestAccuracyScore()

• TestF1Score()

• TestPrecisionByClass()

• TestRecallByClass()

• TestColumnDrift(column_name=target)

• TestNumberOfRows()

• TestLogLoss() - if probabilistic classification

• TestRocAuc() - if probabilistic classification

Optional parameters for target drift:

• stattest

• stattest_threshold

How to set data drift parameters

Binary Classification (Top K) Test Preset

Preset name: BinaryClassificationTopKTestPreset()

Composition:

• TestAccuracyScore(k=k)

• TestPrecisionScore(k=k)

• TestRecallScore(k=k)

• TestF1Score(k=k)

• TestColumnDrift(column_name=target)

• TestRocAuc()

• TestLogLoss()

Required parameters:

• k

Optional parameters:

• stattest

• stattest_threshold

• probas_threshold

How to set data drift parameters

Binary Classification Test Preset

Preset name: BinaryClassificationTestPreset()

Composition:

• TestColumnDrift(column_name=target)

• TestPrecisionScore()

• TestRecallScore()

• TestF1Score()

• TestAccuracyScore()

• TestRocAuc() - if probabilistic classification

Optional parameters:

• stattest

• stattest_threshold

• probas_threshold

How to set data drift parameters

RecSys (Recommender Systems) Test Preset

Preset name: RecsysTestPreset()

Composition:

• TestPrecisionTopK()

• TestRecallTopK()

• TestMAPK()

• TestNDCGK()

• TestHitRateK()

Required parameters:

• k

Optional parameters:

• min_rel_score: Optional[int]

• no_feedback_users: bool

How to set custom Test conditions? Use parameters (e.g. equal, not equal, greater than, etc.) to set Test Conditions.

Data Quality

Data Integrity

Missing Values

Defaults for Missing Values. The metrics that calculate the number or share of missing values detect four types of the values by default: Pandas nulls (None, NAN, etc.), "" (empty string), Numpy "-inf" value, Numpy "inf" value. You can also pass a custom missing values as a parameter and specify if you want to replace the default list. Example:

TestNumberOfMissingValues(missing_values=["", 0, "n/a", -9999, None], replace=True)

Correlations

Column Values

Data Drift

By default, all data drift tests use the Evidently drift detection logic that selects a different statistical test or metric based on feature type and volume. You always need a reference dataset. To modify the logic or select a different test, you should set data drift parameters.

Regression

Defaults for Regression tests: if there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model that predicts the optimal constant (varies by the metric). You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.

Classification

You can apply the tests for non-probabilistic, probabilistic classification, and ranking. The underlying metrics will be calculated slightly differently depending on the provided inputs: only labels, probabilities, decision threshold, and/or K (to compute, e.g., precision@K).

Defaults for Classification tests. If there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model. It is based on a set of heuristics to verify that the quality is better than random. You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.

Ranking and Recommendations

Check individual metric descriptions here.

Optional shared parameters:

• no_feedback_users: bool = False. Specifies whether to include the users who did not select any of the items, when computing the quality metrics. Default: False.

• min_rel_score: Optional[int] = None. Specifies the minimum relevance score to consider relevant when calculating the quality metrics for non-binary targets (e.g., if a target is a rating or a custom score).