Trace contains failed spans
This page explains what the Trace contains failed spans rule checks, when it raises a finding, and how to set its threshold.
Definition
Trace contains failed spans (key trace_error_spans) counts the spans in a trace that ended in error. It raises a finding when that count is above the number you tolerate.
Property | Value |
Unit | trace (one request) |
Check kind | deterministic (no model call) |
Category | Operational thresholds |
Default severity | MEDIUM |
Applies to | Every trace |
The rule passes a trace when its failed spans are at or below the threshold. It fails the trace when they're above it.
Why it matters
A request can return a reply to the user even when something inside it broke. For example, a tool call might fail and the agent might answer anyway. The final answer hides the failure. This rule finds those requests by looking at every span in the trace, not only the result.
What it requires
- Permission: You need permission to change rules in the project. For details, see Project permissions.
- Traces: The rule reads the traces your application already sends. You don't need any extra setup. To start sending traces, see Instrument your code.
- No model key: This is a deterministic check, so it doesn't call a model.
Configuration examples
Create the rule from the template catalog. You'll find it under Operational thresholds, or search for trace_error_spans. To start creating it, select Configure › on its card. For the full procedure, see Create a rule.
On the Configure rule page, the rule has one parameter.
Parameter | Type | Required | Default | Minimum |
Tolerated failed spans | Whole number | Yes |
|
|
The rule fails a trace when it has more failed spans than this number. The read-only How this rule evaluates panel shows this check as trace.error_spans > params.max_error_spans.
The following table shows common settings.
Tolerated failed spans | Result |
| Any failed span fails the trace. This is the usual setting. |
| A trace fails only when two or more spans fail. |
| A trace fails only when six or more spans fail. Use this when your application retries often and some failures are expected. |
When you're done, select Save & enable.
If the value isn't valid, the form shows one of these messages after you try to save:
- "Tolerated failed spans is required."
- "Tolerated failed spans must be a whole number."
- "Tolerated failed spans must be at least 0."
What a finding shows
When a trace fails this rule, the result includes:
- Observed value: The number of failed spans in the trace.
- Excerpt: The first error message in the trace, up to 500 characters.
- Reasoning: A short summary of how many spans failed, such as "2 of 14 spans errored".
To find these results on your traces, see Explore traces.
Related
- Conversation had failed turns catches failed turns across a whole conversation instead of within one request.
- Trace slower than budget is another per-request operational threshold.
- Agent answered without using a tool catches another problem in a request that a successful reply can hide.
Have a suggestion?