List of all tests and test presets available in Evidently.
This is a reference page. You can return here:
- To discover available tests and choose which to include in a custom test suite.
- To understand which parameters you can change for a specific test or preset.
- To verify which tests are included in a test preset.
You can use the menu on the right to navigate the sections. We organize individual tests into groups, e.g. Data Quality, Data Integrity, Regression, etc. Note that these groups do not match the presets with similar names. For example, there are more Data Quality tests than in the
- Name: the name of the test or test preset.
- Description: plain text explanation of the test, or the content of the preset. For tests, we specify whether it applies to the whole dataset or individual columns.
- Parameters: available configurations.
- Required parameters are necessary for calculations, e.g. a column name for a column-level test.
- Optional parameters modify how the underlying metric is calculated, e.g. which statistical test or correlation method is used.
- Test condition parameters help set the conditions (e.g. equal, not equal, greater than, etc.) that define the expectations from the test output. If the condition is violated, the test returns a fail. Here you can see the complete list of the standard condition parameteres. They apply to most of the tests, and are optional.
- Default tests condition: they apply if you do not set a custom сondition.
- With reference: the test conditions that apply when you pass a reference dataset and Evidently can derive expectations from it.
- No reference: the test conditions that apply if you do not provide the reference. They are based on heuristics.
Default conditions for each Test in the Preset match the Test's defaults. You can see them in the tables below. The listed Preset parameters apply to the relevant individual Tests inside the Preset.
Defaults for Data Integrity. If there is no reference data or defined conditions, data integrity will be checked against a set of heuristics. If you pass the reference data, Evidently will automatically derive all relevant statistics (e.g., number of columns, rows, share of missing values etc.) and apply default test conditions. You can also pass custom test conditions.
Defaults for Missing Values. The metrics that calculate the number or share of missing values detect four types of the values by default: Pandas nulls (None, NAN, etc.), "" (empty string), Numpy "-inf" value, Numpy "inf" value. You can also pass a custom missing values as a parameter and specify if you want to replace the default list. Example:
TestNumberOfMissingValues(missing_values=["", 0, "n/a", -9999, None], replace=True)
Defaults for data quality. If there is no reference data or defined conditions, data quality will be checked against a set of heuristics. If you pass the reference data, Evidently will automatically derive all relevant statistics (e.g., min value, max value, value range, value list, etc.) and apply default test conditions. You can also pass custom test conditions.
Defaults for Data Drift. By default, all data drift tests use the Evidently drift detection logic that selects a different statistical test or metric based on feature type and volume. You always need a reference dataset.
Defaults for Regression tests: if there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model that predicts the optimal constant (varies by the metric). You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.
You can apply the tests for non-probabilistic, probabilistic classification, and ranking. The underlying metrics will be calculated slightly differently depending on the provided inputs: only labels, probabilities, decision threshold, and/or K (to compute, e.g., precision@K).
Defaults for Classification tests. If there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model. It is based on a set of heuristics to verify that the quality is better than random. You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.