Installation
RAGAS Evaluators
AnswerCorrectness
Measures answer correctness compared to ground truth using a weighted average of factuality and semantic similarity.ScorerArgs
AnswerRelevancy
Scores the relevancy of the generated answer to the given question. Answers with incomplete, redundant or unnecessary information are penalized.ScorerArgs
AnswerSimilarity
Scores the semantic similarity between the generated answer and ground truth.ScorerArgs
ContextEntityRecall
Estimates context recall by estimating TP and FN using annotated answer and retrieved context.ScorerArgs
ContextPrecision
ContextPrecision evaluator function.ScorerArgs
ContextRecall
ContextRecall evaluator function.ScorerArgs
ContextRelevancy
ContextRelevancy evaluator function.ScorerArgs
Faithfulness
Measures factual consistency of the generated answer with the given context.ScorerArgs
LLM Evaluators
Battle
Test whether an output better performs theinstructions than the original
(expected) value.
ScorerArgs
ClosedQA
Test whether an output answers theinput using knowledge built into the model.
You can specify criteria to further constrain the answer.
ScorerArgs
Factuality
Test whether an output is factual, compared to an original (expected) value.
ScorerArgs
Humor
Test whether an output is funny.ScorerArgs
Possible
Test whether an output is a possible solution to the challenge posed in the input.ScorerArgs
Security
Test whether an output is malicious.ScorerArgs
Sql
Test whether a SQL query is semantically the same as a reference (output) query.ScorerArgs
Summary
Test whether an output is a better summary of theinput than the original (expected) value.
ScorerArgs
Translation
Test whether anoutput is as good of a translation of the input in the specified language
as an expert (expected) value.
ScorerArgs
String Evaluators
EmbeddingSimilarity
A scorer that uses cosine similarity to compare two strings.ScorerArgs
ExactMatch
A simple scorer that tests whether two values are equal. If the value is an object or array, it will be JSON-serialized and the strings compared for equality.reflection
Levenshtein
A simple scorer that uses the Levenshtein distance to compare two strings.reflection
LevenshteinScorer
LevenshteinScorer evaluator function.reflection
JSON Evaluators
JSONDiff
A simple scorer that compares JSON objects, using a customizable comparison method for strings (defaults to Levenshtein) and numbers (defaults to NumericDiff).ScorerArgs
ValidJSON
A binary scorer that evaluates the validity of JSON output, optionally validating against a JSON Schema definition (see https://json-schema.org/learn/getting-started-step-by-step#create).ScorerArgs
Custom Evaluators
LLMClassifierFromSpec
LLMClassifierFromSpec evaluator function.string
reflection
LLMClassifierFromSpecFile
LLMClassifierFromSpecFile evaluator function.string
literal | literal | literal | literal | literal | literal | literal | literal | literal
LLMClassifierFromTemplate
LLMClassifierFromTemplate evaluator function.reflection
OpenAIClassifier
OpenAIClassifier evaluator function.ScorerArgs
buildClassificationTools
buildClassificationTools evaluator function.boolean
array
List Evaluators
ListContains
A scorer that semantically evaluates the overlap between two lists of strings. It works by computing the pairwise similarity between each element of the output and the expected value, and then using Linear Sum Assignment to find the best matching pairs.ScorerArgs
Moderation
Moderation
A scorer that uses OpenAI’s moderation API to determine if AI response contains ANY flagged content.ScorerArgs
Numeric Evaluators
NumericDiff
A simple scorer that compares numbers by normalizing their difference.reflection
Configuration
init
init evaluator function.reflection
Utilities
makePartial
makePartial evaluator function.Scorer
string
normalizeValue
normalizeValue evaluator function.unknown
boolean