Refusal detected

This page explains what the Refusal detected rule checks, when you'd use it, and how to tune its pattern for your product.

Definition

Refusal detected flags a model call when the model declined to answer. It matches each completion against a regular expression of refusal phrasing, such as "I can't" or "I'm sorry". When the completion matches, the rule records a failure.

Each failure shows an excerpt of the completion and the reasoning "completion matches the refusal pattern".

Why it matters

A refusal often looks like a successful call. The model returns text, so nothing errors, but the user didn't get an answer. This rule catches the common shape of a canned refusal without calling a model to judge it. It's a deterministic check with no model calls, so it costs nothing to run on all your traffic.

Watch the refusal rate over time. When it rises, the cause is usually a change to a prompt or a policy, not a change in what users ask.

What it requires

• LLM calls with output: The rule checks only model calls (LLM spans) that returned some text. It records no result for a call with an empty completion. To catch blank completions, use Empty output.
• A refusal pattern: The Refusal phrasing parameter is required and can't be blank. It comes with a default, so the rule works without changes.
• Permission to change rules: To create the rule, you need permission to change rules on the project. For details, see Project permissions.

Parameters

The default pattern covers common English refusal openers, and it ignores case. If your product speaks in its own voice or in another language, change the pattern to match how your model actually phrases a refusal. That works better than putting up with false positives from the default.

Configuration examples

To set up the rule:

1. On the New rule page, enter refusal in Search templates, and then select Configure › on the Refusal detected card.

2. On the Configure rule page, for Refusal phrasing, keep the default or enter your own pattern. Select an example under the field to insert it.
3. Select Save & enable.

For the full procedure, see Create a rule.

The following table shows patterns you can use for Refusal phrasing.

Related

• Empty output: flags model calls that returned a blank completion, which this rule skips.
• Stopped by content filter: flags calls that the provider's own safety filter stopped. That's a different event from the model refusing.
• Banned term in output: matches completions against a pattern of terms they must not mention.

Next steps

• Create a rule
• Rules overview
• Explore traces