Judge Models and Authentication 👩⚖️🔑¶
Simpleval supports several judges, each one with its own authentication method. The way to authenticate with each judge is usually via environment variables or similar implicit methods. There might also be judge-specific details found below.
You can see how each judge type is authenticated below, but in short, most judges are based on Lite LLM, so you can always also refer to its documentation.
Tip
You can view supported model ids for Lite LLM judges by running:
1 | |
Anthropic Judge
Required environment variable:
1 | |
Example:
1 | |
See the Anthropic docs for more details.
Azure Judge
Required environment variables:
1 2 3 | |
Example:
1 2 3 | |
Available Versions
You can see available versions in the Azure docs
Bedrock Claude Sonnet Judge
This judge is NOT based on Lite LLM, it is implemented natively with boto3 calling Bedrock.
Required credentials:
AWS credentials must be available, either via environment variables or in a ~/.aws/credentials file.
You can learn more about using AWS credentials in the AWS docs
Gemini Judge
Required environment variable:
1 | |
See the Gemini API docs for more details.
Generic Bedrock Judge
Required credentials:
AWS credentials must be available, either via environment variables or in a ~/.aws/credentials file.
You can learn more about using AWS credentials in the AWS docs
LiteLLM Structured Output Judge
Required authentication:
Depends on the provider you use with LiteLLM.
You can learn about the available providers in the LiteLLM documentation.
And about structured output in LiteLLM here
OpenAI Judge
Required environment variable:
1 | |
See the OpenAI API docs for more details.
Vertex AI Judge
Required environment variable:
1 2 3 | |
Example:
1 2 3 | |
See the Vertex AI docs for more details.