For an intro, read about Core Concepts and check the LLM Quickstart. For a reference code example, see this Descriptor cookbook.
Deterministic evals
Programmatic and heuristics-based evaluations.Pattern match
Check for general pattern matching.Name | Description | Parameters |
---|---|---|
ExactMatch() |
| Required:
|
RegExp() |
| Required:
|
BeginsWith() |
| Required:
|
EndsWith() |
| Required:
|
Content checks
Verify presence of specific words, items or components.Name | Description | Parameters |
---|---|---|
Contains() |
| Required:
|
DoesNotContain() |
| Required:
|
IncludesWords() |
| Required:
|
ExcludesWords() |
| Required:
|
ItemMatch() |
| Required:
|
ItemNoMatch() |
| Required:
|
WordMatch() |
| Required:
|
WordNoMatch() |
| Required:
|
ContainsLink() |
| Optional:
|
Syntax validation
Validate structured data formats or code syntax.Name | Description | Parameters |
---|---|---|
IsValidJSON() |
| Optional:
|
JSONSchemaMatch() |
| Required:
|
JSONMatch() |
| Required:
|
IsValidPython() |
| Optional:
|
IsValidSQL() |
| Optional:
|
Text stats
Descriptive text statistics.Name | Descriptor | Parameters |
---|---|---|
TextLength() |
| Optional:
|
OOVWordsPercentage() |
| Optional:
|
NonLetterCharacterPercentage() |
| Optional:
|
SentenceCount() |
| Optional:
|
WordCount() |
| Optional:
|
Custom
Implement your own programmatic checks.Name | Descriptor | Parameters |
---|---|---|
CustomDescriptor() |
| Optional:
|
CustomColumnsDescriptor() |
| Optional:
|
LLM-based evals
Using an external LLMs with an evaluation prompt. You can specify the LLM to use as an evaluator.Custom
LLM judge templates.Name | Descriptor | Parameters |
---|---|---|
LLMEval() |
| Optional:
|
RAG
RAG-specific evals for retrieval and generation. (Tutorial).Name | Descriptor | Parameters |
---|---|---|
ContextQualityLLMEval() |
| Required:
|
ContextRelevance() |
| Required:
|
FaithfulnessLLMEval() |
| Required:
|
CompletenessLLMEval() |
| Required:
|
Generation
Evals for varied generation scenarios.Name | Descriptor | Parameters |
---|---|---|
CorrectnessLLMEval() |
| Required:
|
DeclineLLMEval() |
| Optional:
|
PIILLMEval() |
| Optional:
|
NegativityLLMEval() |
| Optional:
|
BiasLLMEval() |
| Optional:
|
ToxicityLLMEval() |
| Optional:
|
ML-based evals
Use pre-trained machine learning or embedding models.Name | Descriptor | Parameters |
---|---|---|
SemanticSimilarity() |
| Required:
|
BERTScore() |
| Required:
|
Sentiment() |
| Optional:
|
HuggingFace() |
| Optional:
|
HuggingFaceToxicity() |
| Optional:
|