Acceldata
AIO

Last updated: Nov 16, 2026 15:11 UTC

This page shows how to score AI-generated responses with the Python SDK, one prompt and response at a time or as a whole dataset, and how to read the scores you get back.

You run evals from the Python SDK. The Evals page in the web UI doesn't have any actions.

Prerequisites

  • The Python SDK is installed. For more information, see Instrument your code.
  • You have an API key for your project. For more information, see Create an API key.
  • You know the URL of your AIO server.

Set your credentials

Eval calls need your API key and your server URL. The simplest way to provide them is with environment variables:

export AIO_API_KEY="aio-fake-key-0000"
export AIO_URL="https://aio.example.com"

Replace both values with your own. You can also pass them on each call with the acceldata_aio_tracer_api_key and acceldata_aio_tracer_url parameters. An argument you pass takes priority over the environment variable. The SDK removes a trailing / from the URL.

If the SDK can't find a key or a URL, the call raises a ValueError before it sends anything.

Choose eval types

Each eval type is one kind of evaluation that the server runs. Eval types can be built-in or custom. To see which ones your server has, call get_eval_types():

import acceldata_aio_tracer

for eval_type in acceldata_aio_tracer.get_eval_types():
    print(eval_type.id, eval_type.label, eval_type.enabled, eval_type.is_custom)

Each eval type has these fields:

Field What it holds
id The ID to pass in eval_types
label The display name
description What the eval type checks
enabled Whether the eval type is turned on
is_custom Whether it's a custom eval type rather than a built-in one
threshold_score The eval type's threshold score, when it has one

Unlike eval(), get_eval_types() raises its errors. For the list, see Troubleshooting.

Evaluate a prompt and response

  1. Call acceldata_aio_tracer.eval() with the prompt and the response you want to score.
  2. Read the result it returns.
import acceldata_aio_tracer

enabled_types = [t.id for t in acceldata_aio_tracer.get_eval_types() if t.enabled]

result = acceldata_aio_tracer.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Paris.",
    eval_types=enabled_types,
)

print("Passed:", result.passed)
for evaluation in result.evaluations:
    print(evaluation.type, evaluation.score, evaluation.verdict)

You can also use these optional parameters:

  • eval_types: a list of eval type IDs to run.
  • contexts: a list of your own context strings to send with the pair.
  • run_id: a label that groups evaluations together.
  • attributes: a dictionary of attributes to send with the pair. The SDK already sends attributes from OTEL_RESOURCE_ATTRIBUTES, service.name from OTEL_SERVICE_NAME, and deployment.environment from AIO_ENVIRONMENT or OTEL_DEPLOYMENT_ENVIRONMENT. Values you pass here override them.
  • print_results: set it to False to stop the printed summary. It's True by default.

For every parameter, see Python SDK.

Each request times out after 120 seconds. If the server returns HTTP 429, returns HTTP 500 or higher, or can't be reached, the SDK retries the request once, right away.

Read the scores

eval() returns a result with these fields:

Field What it holds
success True if the call succeeded
evaluations One entry for each eval type, with type, score, verdict, classification, and explanation
passed True when the call succeeded and no evaluation has the verdict yes
failed_evals The evaluations whose verdict is yes
context_applied Which rules matched, which context entities were used, and how many of your own contexts were counted
metadata Metadata the server returned
error The error message when success is False

A verdict of yes means the eval type found an issue. The SDK compares verdicts without regard to case. score is a decimal number. It's 0.0 when the server doesn't return one.

Read the printed summary

By default, eval() prints a summary to standard error. It's in color only when standard error is a terminal. You can also get the same text by calling result.summary().

The summary starts with Acceldata AIO Offline Evaluation — PASSED or Acceldata AIO Offline Evaluation — FAILED. Next comes one line for each evaluation, which shows its type, its score to two decimal places, its verdict, and its classification. A ✓ marks an evaluation that found no issue, and a ✗ marks one whose verdict is yes. When an evaluation has an explanation, its first 200 characters appear on the next line.

When they apply, the summary ends with these lines:

  • The number of context entities used and the number of rules they came from.
  • The number of your own contexts.
  • The model and the run ID.

If the call failed, the summary is a single line, for example Evaluation failed: Authentication failed. Check your Acceldata AIO API key.

Evaluate a dataset

  1. Build a list of dictionaries. Each one needs a prompt string and a response string.
  2. Call acceldata_aio_tracer.eval_batch() with the list.
  3. Read the batch result it returns.
import acceldata_aio_tracer

dataset = [
    {
        "prompt": "What is the capital of France?",
        "response": "The capital of France is Paris.",
    },
    {
        "prompt": "What is 2 + 2?",
        "response": "2 + 2 equals 4.",
        "contexts": ["Basic arithmetic facts."],
    },
]

batch = acceldata_aio_tracer.eval_batch(dataset)

print("Run ID:", batch.run_id)
print("Pass rate:", batch.pass_rate)
for item in batch.results:
    print(item.passed, [e.type for e in item.failed_evals])

Each item can also have its own contexts, eval_types, and metadata. An item's eval_types takes priority over the eval_types you pass to eval_batch().

When the batch runs:

  • The SDK evaluates five items at a time by default. To change that, set max_concurrent.
  • Every item shares one run_id. If you don't pass one, the SDK makes one that starts with batch_, followed by 12 hexadecimal characters.
  • Results come back in the same order as your dataset.
  • If one item fails, the failure is recorded for that item, and the rest of the batch keeps running.

Before it sends any requests, the SDK checks the dataset. If the dataset is empty or an item is invalid, the call raises an error. For the messages, see Troubleshooting.

Read batch scores

The batch result has these fields:

Field What it holds
results One result for each item, in dataset order, with the same fields as a single eval() result
run_id The run ID shared by every item
all_passed True only when the batch has items and every item passed
pass_rate The fraction of items that passed, or 0.0 for an empty batch

Instead of printing a summary for every item, eval_batch() prints one summary for the whole batch. To get that text yourself, call batch.aggregate_summary(). The summary starts with Acceldata AIO Batch Evaluation — ALL PASSED or with the number of failures, for example Acceldata AIO Batch Evaluation — 2 FAILED. It then shows the totals and the pass rate, for example Total: 10 Passed: 8 Failed: 2 Rate: 80%, followed by the run ID. Then comes one line for each item. An item whose call succeeded shows its eval types. An item that couldn't be evaluated shows its error and counts as failed.

Troubleshooting

eval() doesn't raise errors when a request fails. Instead, it returns success as False and puts the message in error. The following table lists the messages.

Message What it means
Authentication failed. Check your Acceldata AIO API key. The server rejected the API key. The SDK doesn't retry.
Request timed out after 120s. The evaluation may still be running on the server. The server didn't respond within 120 seconds. The SDK doesn't retry.
Cannot connect to Acceldata AIO server at https://aio.example.com, followed by connection details The SDK couldn't reach the server, even after one retry. Check AIO_URL.
Server returned non-JSON response (HTTP 502) The server sent back something that isn't JSON.
The server's error message, or a status such as HTTP 404 The server returned some other error.

eval_batch() raises these errors before it sends any requests:

Error Cause
dataset must be a non-empty list of prompt/response dicts The dataset is empty or isn't a list.
dataset[0] must be a dict, got str An item isn't a dictionary.
dataset[1] must have a 'prompt' string key An item has no prompt string.
dataset[1] must have a 'response' string key An item has no response string.

get_eval_types() raises these errors:

Error Cause
Authentication failed. Check your Acceldata AIO API key. The server rejected the API key.
Cannot connect to Acceldata AIO server at https://aio.example.com, followed by connection details The SDK couldn't reach the server.
Request to Acceldata AIO server timed out after 120s The server didn't respond within 120 seconds.
Failed to fetch eval types, followed by the error details The request failed for another reason.

Next steps