Conversation over token budget

This page explains what the Conversation over token budget rule checks, when it raises a finding, and how to set its budget.

Definition

Conversation over token budget (session_token_budget) checks each conversation and fails it when the conversation's total token count is higher than the budget you set.

The rule applies to every conversation. It fails a conversation when its total tokens are greater than Token budget. A conversation that uses exactly the budget passes.

Because it's a deterministic check, it doesn't call a model.

Why it matters

Cost builds up over a whole conversation, not over one call. A runaway loop, where the application keeps calling the model turn after turn, often shows up first as a conversation that uses far more tokens than usual. Each single call can look normal while the conversation as a whole is too expensive.

When a conversation fails, the reasoning tells you how far over budget it went, for example: "conversation used 132450 tokens, budget 100000".

What it requires

• Conversations in your telemetry. The rule evaluates sessions, so your traces need session identifiers. To set them, see Track users and sessions.
• Token usage on your calls. If no token count is recorded, the conversation's total reads as 0. Then the rule never fails it.
• Permission to change rules. To create the rule, you need permission to change rules in the project. For details, see Project permissions.

Configuration examples

The rule has one parameter.

The rule fails the conversation when its total tokens are greater than this value. The form suggests 50000, 100000, and 500000. Select an example to put it in the field.

To set up the rule:

1. In your project, go to Rules and select + New rule.
2. On the New rule page, enter session_token_budget in Search templates, or select Operational thresholds in the Category list. Then on the Conversation over token budget card, select Configure ›.

3. On the Configure rule page, for Token budget, enter the most tokens a single conversation should use, for example 50000.

4. Select Save & enable.

For the other fields on this page, see Create a rule.

If the value isn't valid, the form shows one of these messages after you save:

• "Token budget is required."
• "Token budget must be a whole number."
• "Token budget must be at least 1."

A conversation keeps growing as users send more turns. When more of a conversation arrives, the rule can check it again. The new result replaces the earlier one for the same rule version.

To find conversations that failed this rule, use the Rules column and filter on the conversation list. For details, see Explore traces.

Related

• LLM call over token budget catches one expensive model call.
• Request over token budget catches one request whose calls together use too many tokens.
• Conversation ran long catches conversations with more turns than expected.
• Conversation had failed turns catches conversations where a turn errored.

Next steps

• Create a rule
• Track users and sessions
• Cost tracking
• Rules overview