Agent answered without using a tool

This page explains what the Agent answered without using a tool rule checks, when it fires, and how to set it up for a project.

Definition

Agent answered without using a tool flags a request (trace) that made at least one large language model (LLM) call but never called a tool.

The rule checks traces that have at least one LLM call. Traces with no LLM calls get no result: they don't pass and they don't fail.

A trace fails when its tool call count is zero. A trace with one or more tool calls passes.

When a trace fails, its result shows:

• Observed value: the number of LLM calls in the trace.
• Reasoning: a sentence such as "3 LLM call(s) and no tool call".

Why it matters

In a retrieval or agent workflow, the model is supposed to look things up or act through tools before it answers. If it answers without calling any tool, it's answering from memory instead of from your data. The reply can look normal, so this kind of failure is easy to miss without a check.

What it requires

• Traces with LLM calls. The rule ignores traces that contain no LLM call.
• A project where tool use is expected. The rule fails every request that answers without a tool. In a project where some requests normally answer without tools, each of those requests becomes a finding. The rule watches every environment in its project.
• Permission to change rules. To create the rule, you need permission to modify rules on the project. For details, see Project permissions.

No model calls are involved. This is a deterministic check, so it doesn't need a judge or classifier model.

Configuration examples

The rule has no parameters. When you configure it, the parameters area says the template has nothing to configure. You only set the name, project, and severity.

Watch an agent project with the default severity

1. In your project, go to Rules and select + New rule.
2. On the New rule page, enter trace_no_tool_call in Search templates. Or, in the Category rail, select Tool and agent structure.

3. On the Agent answered without using a tool card, select Configure ›.
4. For Name, keep the suggested name, such as "Agent answered without using a tool — support-agent", or enter your own. The name must be unique across your whole organization, not just this project.
5. For Project, select the project that runs your agent or retrieval workflow.
6. For Severity, keep MEDIUM.

7. Select Save & enable.

The rule is enabled as soon as you save it. For the full set of fields and the validation messages, see Create a rule.

Treat answers from memory as high severity

When an answer that skips your data is a serious problem, for example in a support agent that must quote your knowledge base, follow the previous steps, but for Severity, select HIGH or CRITICAL. Severity is what alerting routes on, so a higher severity makes these findings stand out.

Find the traces that failed

After the rule runs, open your project's trace list. In the Rules filter, select this rule to list only the traces it failed, most recent failure first. For more on reading results, see Explore traces.

Related

• Groundedness: checks whether an answer is supported by the documents it retrieved.
• Trace contains failed spans: catches a request where something inside failed even though the user got a reply.
• Request over token budget: catches an agent loop that runs up tokens across many calls.

Next steps

• Create a rule
• Rules overview
• Explore traces
• Project permissions