Acceldata
AIO

Last updated: Nov 16, 2026 15:07 UTC

This page explains what evals measure in AIO, what a score and a verdict mean, and when an eval runs.

Evals score AI-generated text. You send a prompt and the response your application produced, and AIO scores the response for each eval type you ask for. Use evals to check responses before you ship a change, or to test a whole dataset of prompts and responses at once.

You run evals from the Python SDK. The Evals page in the AIO UI doesn't show runs or scores.

What an eval measures

Each eval measures one thing, called an eval type. AIO has built-in eval types, and your organization can add custom ones. To see which eval types your AIO server offers, call get_eval_types() from the Python SDK. It returns each type's ID, label, and description, and tells you whether the type is enabled and whether it's custom.

For each eval type, the result gives you:

  • Score: a number for the eval type.
  • Verdict: whether the eval type found an issue. A verdict of yes means it found one.
  • Classification: the category the eval type assigned to the response.
  • Explanation: why the eval type gave that result.

A response passes when the eval call succeeds and no eval type returns a verdict of yes. When you evaluate a dataset, the result also gives you a pass rate: the share of items that passed.

When evals run

An eval runs when you call it from your code. Run evals in two ways:

  • One prompt and response: call eval() with a prompt and a response.
  • A dataset: call eval_batch() with a list of prompt and response pairs. The SDK evaluates the items in parallel, and every item shares one run ID that groups the evaluations together. If you don't pass a run ID, one is created for you.

How it works

  1. Your code sends a prompt and a response, or a dataset of them, with eval() or eval_batch(). The SDK authenticates with your project's API key.
  2. The AIO server runs each eval type you asked for. It uses the rules and context set up in AIO, and you can send extra context with the request.
  3. AIO returns a score, verdict, classification, and explanation for each eval type. The result also shows which rules matched and how much context was applied.
  4. By default, the SDK prints a summary of the results, showing whether each response passed or failed.

Add your own scores and feedback

Evals aren't the only way to score a response. If you score responses yourself, or collect feedback from your users, attach those scores to the matching span in a trace with log_score(). A score can be a number, a boolean such as a thumbs-up, or a text label such as accurate. You can also attach a score to a past trace when feedback arrives later. To learn about traces and spans, see Tracing overview.

Next steps