You can build fully custom Metrics/Tests to handle any column- or dataset-level evaluations. This lets you implement business metrics, weighted scores, etc.
There are ways to customize your evals that do not require creating Metrics from scratch:
Creating a custom Metric involves:
  • (Required). Implementing the Metric calculation method.
  • (Optional). Defining the default Test conditions that apply when you run Tests for this Metric (with or without Reference) without passing a custom condition.
  • (Optional). Creating a custom visualization for this Metric using Plotly. If you skip this, the Metric will appear as a simple counter in the Report.
Once you implement the Metric, you can use it as usual: include in Reports, view in the Evidently Cloud (or a self-hosted UI), and visualize over time on the Dashboard.

​
Example implementation

This is advanced functionality that assumes you’re comfortable working with the codebase. Refer to existing metrics for examples. To implement the visualization, you must be familiar with Plotly.
Let’s implement MyMaxMetric which calculates the maximum value in a column. Imports: 
import pandas as pd
import numpy as np
from evidently import Report
from evidently import Dataset
from evidently import DataDefinition
from evidently.core.report import Context
from evidently.core.metric_types import SingleValue
from evidently.core.metric_types import SingleValueMetric
from evidently.core.metric_types import SingleValueCalculation
from evidently.core.metric_types import BoundTest
from evidently.tests import Reference, eq

from evidently.legacy.renderers.html_widgets import plotly_figure

from typing import Optional
from typing import List
from plotly.express import line
Implementation: 
class MyMaxMetric(SingleValueMetric):
    column: str

    def _default_tests(self) -> List[BoundTest]:
        return [eq(0).bind_single(self.get_fingerprint())]

    def _default_tests_with_reference(self) -> List[BoundTest]:
        return [eq(Reference(relative=0.1)).bind_single(self.get_fingerprint())]

# implementation
class MaxMetricImplementation(SingleValueCalculation[MyMaxMetric]):
    def calculate(self, context: Context, current_data: Dataset, reference_data: Optional[Dataset]) -> SingleValue:
        x = current_data.column(self.metric.column).data
        value = x.max()
        result = self.result(value=value)
        figure = line(x)
        figure.add_hrect(6, 10)
        result.widget = [plotly_figure(title=self.display_name(), figure=figure)] #skip this to get a simple counter
        return result

    def display_name(self) -> str:
        return f"Max value for {self.metric.column}"
The default Test will checks if the max value is 0 (or within ±10% of the reference value). This applies if you invoke the Tests without setting a custom threshold. This implementation uses the default (counter) render. Alternatively, you can define the widget as a Plotly figure. In this case, set the result.widget as shown in the code.

​
Example use

Once implemented, you can reference your custom Metric in a Report as usual. Let’s create a sample toy dataset: 
data = {
    "Item": [f"Item_{i}" for i in range(1, 11)],
    "Quantity": np.random.randint(1, 50, size=10),
    "Sales": np.random.uniform(100, 5000, size=10).round(2),
}

df = pd.DataFrame(data)

dataset = Dataset.from_pandas(
    pd.DataFrame(df),
    data_definition=DataDefinition()
)
Add my MyMaxMetric to the Report: 
report = Report([
    MyMaxMetric(column="Sales")
])
my_eval = report.run(dataset, None)
my_eval
Want a Metric added to the core library? Share your idea or feature request by opening a GitHub issue.