List of all Tests and Test Presets available in Evidently.
How to use this page
This is a reference page. You can return here:
To discover available Tests and choose which to include in a custom Test suite.
To understand which parameters you can change for a specific Test or Preset.
To verify which tests are included in a Test Preset.
You can use the menu on the right to navigate the sections.
How to read the tables
Name: the name of the Test or Test preset.
Description: plain text explanation. For Tests, we specify whether it applies to the whole dataset or individual columns.
Parameters: available configurations.
Required parameters are necessary for calculations, e.g. a column name for a column-level test.
Optional parameters modify how the underlying metric is calculated, e.g. which statistical test or correlation method is used.
Test condition parameters set the conditions (e.g. equal, not equal, greater than, etc.) that define the expectations from the Test output. If the condition is violated, the test returns a fail. They apply to most of the Tests, and are optional.
Default Test condition: they apply if you do not set a custom сondition.
With reference: the Test conditions that apply when you pass a reference dataset and Evidently can derive conditions from it. (E.g. expect +/- 10% from reference).
No reference: the Test conditions that apply if you do not provide the reference. They are based on heuristics.
Test visualizations. Each Test also includes a default render. If you want to see the visualization, navigate to the example notebooks.
We are doing our best to maintain this page up to date. In case of discrepancies, consult the API reference or the "All tests" notebook in the Examples section. If you notice an error, please send us a pull request to update the documentation!
Test Presets
Default conditions for each Test in the Preset match the Test's defaults. You can see them in the tables below. The listed Preset parameters apply to the relevant individual Tests inside the Preset.
NoTargetPerformance Test Preset
Preset name: NoTargetPerformanceTestPreset()
Composition:
TestShareOfDriftedColumns()
TestColumnDrift(column_name=prediction)
TestColumnShareOfMissingValues() for all or сolumns if provided
TestShareOfOutRangeValues() for all numerical or specified columns
TestShareOfOutListValues() for all categorical or specified columns
TestMeanInNSigmas() for all numerical or specified columns
How to set custom Test conditions? Use parameters (e.g. equal, not equal, greater than, etc.) to set Test Conditions.
Data Quality
Data Integrity
Missing Values
Defaults for Missing Values. The metrics that calculate the number or share of missing values detect four types of the values by default: Pandas nulls (None, NAN, etc.), "" (empty string), Numpy "-inf" value, Numpy "inf" value. You can also pass a custom missing values as a parameter and specify if you want to replace the default list. Example:
By default, all data drift tests use the Evidently drift detection logic that selects a different statistical test or metric based on feature type and volume. You always need a reference dataset. To modify the logic or select a different test, you should set data drift parameters.
Regression
Defaults for Regression tests: if there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model that predicts the optimal constant (varies by the metric). You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.
Classification
You can apply the tests for non-probabilistic, probabilistic classification, and ranking. The underlying metrics will be calculated slightly differently depending on the provided inputs: only labels, probabilities, decision threshold, and/or K (to compute, e.g., precision@K).
Defaults for Classification tests. If there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model. It is based on a set of heuristics to verify that the quality is better than random. You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.
no_feedback_users: bool = False. Specifies whether to include the users who did not select any of the items, when computing the quality metrics. Default: False.
min_rel_score: Optional[int] = None. Specifies the minimum relevance score to consider relevant when calculating the quality metrics for non-binary targets (e.g., if a target is a rating or a custom score).