Custom Text Descriptor

To run a check not available in Evidently, you can implement it as a custom function. Use this for building your own programmatic evaluators.

You can also customize existing evals with parameters, such as defining custom LLM judges or using regex-based metrics like Contains for word lists. See available descriptors.

Pre-requisites:

You know how to use built-in descriptors.

Imports

import pandas as pd

from evidently import Dataset, DataDefinition
from evidently.core.datasets import DatasetColumn
from evidently.descriptors import CustomColumnDescriptor, CustomDescriptor

Toy data to run the example

To generate toy data and create a Dataset object:

data = [
    ["Can fish fly?", "no", ""],
    ["Is the sky blue?", "yes", "yes"],
    ["Is milk liquid??", "yes", "yes"]
]

columns = ["question", "target_answer", "answer"]

df = pd.DataFrame(data, columns=columns)

eval_df = Dataset.from_pandas(
    df,
    data_definition=DataDefinition())

Single column check

You can define a CustomColumnDescriptor that will:

take any column from your dataset to evaluate each value inside it
return a single column with numerical (num) scores or categorical (cat) labels.

Implement it as a Python function that takes a Pandas Series as input and return a transformed Series. For example, to check if the column is empty:

def is_empty(data: DatasetColumn) -> DatasetColumn:
    return DatasetColumn(
        type="cat",
        data=pd.Series([
            "EMPTY" if val == "" else "NON EMPTY"
            for val in data.data]))

To use this descriptor on your data:

eval_df.add_descriptors(descriptors=[
    CustomColumnDescriptor("answer", is_empty, alias="is_empty"),
])

Publish to a dataframe:

eval_df.as_dataframe()

Multi-column check

You can alternatively define a CustomDescriptor that:

Takes one or many named columns from your dataset,
Returns one or many transformed columns.

Pairwise evaluation. For example, to check exact match between target_answer and answer columns, and return a label:

def exact_match(dataset: Dataset) -> DatasetColumn:
    return DatasetColumn(
        type="cat",
        data=pd.Series([
            "MATCH" if val else "MISMATCH"
            for val in dataset.column("target_answer").data
            == dataset.column("answer").data]))

To use this descriptor on your data:

eval_df.add_descriptors(descriptors=[
    CustomDescriptor(exact_match, alias="exact"),
])

Multiple scores. You can also use CustomDescriptor to run evals for multiple columns and return multiple scores. As a fun example, let’s reverse all words in the question and answer columns:

from typing import Union, Dict

def reverse_text(dataset: Dataset) -> Union[DatasetColumn, Dict[str, DatasetColumn]]:
    return {
        "reversed_question": DatasetColumn(
            type="cat",
            data=pd.Series([
                value[::-1] for value in dataset.column("question").data])),
        "reversed_answer": DatasetColumn(
            type="cat",
            data=pd.Series([
                value[::-1] for value in dataset.column("answer").data]))}

To use this descriptor on your data:

eval_df.add_descriptors(descriptors=[
    CustomDescriptor(reverse_text),
])

Reference

Metric Customization

Explainers

Imports

Single column check

Multi-column check

​Imports

​Single column check

​Multi-column check

Imports

Single column check

Multi-column check