This page shows how to score AI-generated responses with the Python SDK, one prompt and response at a time or as a whole dataset, and how to read the scores you get back.
You run evals from the Python SDK. The Evals page in the web UI doesn't have any actions.
Prerequisites
- The Python SDK is installed. For more information, see Instrument your code.
- You have an API key for your project. For more information, see Create an API key.
- You know the URL of your AIO server.
Set your credentials
Eval calls need your API key and your server URL. The simplest way to provide them is with environment variables:
export AIO_API_KEY="aio-fake-key-0000"
export AIO_URL="https://aio.example.com"
Replace both values with your own. You can also pass them on each call with the acceldata_aio_tracer_api_key and acceldata_aio_tracer_url parameters. An argument you pass takes priority over the environment variable. The SDK removes a trailing / from the URL.
If the SDK can't find a key or a URL, the call raises a ValueError before it sends anything.
Choose eval types
Each eval type is one kind of evaluation that the server runs. Eval types can be built-in or custom. To see which ones your server has, call get_eval_types():
import acceldata_aio_tracer
for eval_type in acceldata_aio_tracer.get_eval_types():
print(eval_type.id, eval_type.label, eval_type.enabled, eval_type.is_custom)
Each eval type has these fields:
| Field | What it holds |
|---|---|
id |
The ID to pass in eval_types |
label |
The display name |
description |
What the eval type checks |
enabled |
Whether the eval type is turned on |
is_custom |
Whether it's a custom eval type rather than a built-in one |
threshold_score |
The eval type's threshold score, when it has one |
Unlike eval(), get_eval_types() raises its errors. For the list, see Troubleshooting.
Evaluate a prompt and response
- Call
acceldata_aio_tracer.eval()with the prompt and the response you want to score. - Read the result it returns.
import acceldata_aio_tracer
enabled_types = [t.id for t in acceldata_aio_tracer.get_eval_types() if t.enabled]
result = acceldata_aio_tracer.eval(
prompt="What is the capital of France?",
response="The capital of France is Paris.",
eval_types=enabled_types,
)
print("Passed:", result.passed)
for evaluation in result.evaluations:
print(evaluation.type, evaluation.score, evaluation.verdict)
You can also use these optional parameters:
eval_types: a list of eval type IDs to run.contexts: a list of your own context strings to send with the pair.run_id: a label that groups evaluations together.attributes: a dictionary of attributes to send with the pair. The SDK already sends attributes fromOTEL_RESOURCE_ATTRIBUTES,service.namefromOTEL_SERVICE_NAME, anddeployment.environmentfromAIO_ENVIRONMENTorOTEL_DEPLOYMENT_ENVIRONMENT. Values you pass here override them.print_results: set it toFalseto stop the printed summary. It'sTrueby default.
For every parameter, see Python SDK.
Each request times out after 120 seconds. If the server returns HTTP 429, returns HTTP 500 or higher, or can't be reached, the SDK retries the request once, right away.
Read the scores
eval() returns a result with these fields:
| Field | What it holds |
|---|---|
success |
True if the call succeeded |
evaluations |
One entry for each eval type, with type, score, verdict, classification, and explanation |
passed |
True when the call succeeded and no evaluation has the verdict yes |
failed_evals |
The evaluations whose verdict is yes |
context_applied |
Which rules matched, which context entities were used, and how many of your own contexts were counted |
metadata |
Metadata the server returned |
error |
The error message when success is False |
A verdict of yes means the eval type found an issue. The SDK compares verdicts without regard to case. score is a decimal number. It's 0.0 when the server doesn't return one.
Read the printed summary
By default, eval() prints a summary to standard error. It's in color only when standard error is a terminal. You can also get the same text by calling result.summary().
The summary starts with Acceldata AIO Offline Evaluation — PASSED or Acceldata AIO Offline Evaluation — FAILED. Next comes one line for each evaluation, which shows its type, its score to two decimal places, its verdict, and its classification. A ✓ marks an evaluation that found no issue, and a ✗ marks one whose verdict is yes. When an evaluation has an explanation, its first 200 characters appear on the next line.
When they apply, the summary ends with these lines:
- The number of context entities used and the number of rules they came from.
- The number of your own contexts.
- The model and the run ID.
If the call failed, the summary is a single line, for example Evaluation failed: Authentication failed. Check your Acceldata AIO API key.
Evaluate a dataset
- Build a list of dictionaries. Each one needs a
promptstring and aresponsestring. - Call
acceldata_aio_tracer.eval_batch()with the list. - Read the batch result it returns.
import acceldata_aio_tracer
dataset = [
{
"prompt": "What is the capital of France?",
"response": "The capital of France is Paris.",
},
{
"prompt": "What is 2 + 2?",
"response": "2 + 2 equals 4.",
"contexts": ["Basic arithmetic facts."],
},
]
batch = acceldata_aio_tracer.eval_batch(dataset)
print("Run ID:", batch.run_id)
print("Pass rate:", batch.pass_rate)
for item in batch.results:
print(item.passed, [e.type for e in item.failed_evals])
Each item can also have its own contexts, eval_types, and metadata. An item's eval_types takes priority over the eval_types you pass to eval_batch().
When the batch runs:
- The SDK evaluates five items at a time by default. To change that, set
max_concurrent. - Every item shares one
run_id. If you don't pass one, the SDK makes one that starts withbatch_, followed by 12 hexadecimal characters. - Results come back in the same order as your dataset.
- If one item fails, the failure is recorded for that item, and the rest of the batch keeps running.
Before it sends any requests, the SDK checks the dataset. If the dataset is empty or an item is invalid, the call raises an error. For the messages, see Troubleshooting.
Read batch scores
The batch result has these fields:
| Field | What it holds |
|---|---|
results |
One result for each item, in dataset order, with the same fields as a single eval() result |
run_id |
The run ID shared by every item |
all_passed |
True only when the batch has items and every item passed |
pass_rate |
The fraction of items that passed, or 0.0 for an empty batch |
Instead of printing a summary for every item, eval_batch() prints one summary for the whole batch. To get that text yourself, call batch.aggregate_summary(). The summary starts with Acceldata AIO Batch Evaluation — ALL PASSED or with the number of failures, for example Acceldata AIO Batch Evaluation — 2 FAILED. It then shows the totals and the pass rate, for example Total: 10 Passed: 8 Failed: 2 Rate: 80%, followed by the run ID. Then comes one line for each item. An item whose call succeeded shows its eval types. An item that couldn't be evaluated shows its error and counts as failed.
Troubleshooting
eval() doesn't raise errors when a request fails. Instead, it returns success as False and puts the message in error. The following table lists the messages.
| Message | What it means |
|---|---|
Authentication failed. Check your Acceldata AIO API key. |
The server rejected the API key. The SDK doesn't retry. |
Request timed out after 120s. The evaluation may still be running on the server. |
The server didn't respond within 120 seconds. The SDK doesn't retry. |
Cannot connect to Acceldata AIO server at https://aio.example.com, followed by connection details |
The SDK couldn't reach the server, even after one retry. Check AIO_URL. |
Server returned non-JSON response (HTTP 502) |
The server sent back something that isn't JSON. |
The server's error message, or a status such as HTTP 404 |
The server returned some other error. |
eval_batch() raises these errors before it sends any requests:
| Error | Cause |
|---|---|
dataset must be a non-empty list of prompt/response dicts |
The dataset is empty or isn't a list. |
dataset[0] must be a dict, got str |
An item isn't a dictionary. |
dataset[1] must have a 'prompt' string key |
An item has no prompt string. |
dataset[1] must have a 'response' string key |
An item has no response string. |
get_eval_types() raises these errors:
| Error | Cause |
|---|---|
Authentication failed. Check your Acceldata AIO API key. |
The server rejected the API key. |
Cannot connect to Acceldata AIO server at https://aio.example.com, followed by connection details |
The SDK couldn't reach the server. |
Request to Acceldata AIO server timed out after 120s |
The server didn't respond within 120 seconds. |
Failed to fetch eval types, followed by the error details |
The request failed for another reason. |

Have a suggestion?