Expected format absent

This page shows what the Expected format absent rule checks, what it needs, and how to set its pattern so you catch completions that come back in the wrong shape.

Definition

Expected format absent checks each model call's completion against a pattern you write. The rule fails when the completion doesn't match. Use it for any shape the caller depends on, such as a required heading, a table, or a fixed prefix.

The rule only checks model calls that returned text. A blank completion isn't checked, so it never fails this rule. To catch blank completions, use Empty output.

When a completion fails, the result shows an excerpt of the completion and a summary that names your pattern, for example "completion does not match the expected shape ^## ".

Why it matters

Code that parses a model's answer often expects a fixed shape. When the shape is missing, the call still reports success, but the next step in your application can break or show the user a broken answer. This rule finds those calls, even when nothing else went wrong.

It's a deterministic check, so it makes no model calls of its own.

What it requires

• Traces that contain model calls with completion text. To send traces, see Instrument your code.
• Permission to change rules in the project. See Project permissions.
• A value for the following parameter. It has no default, so you have to enter one before you can save.

The pattern uses RE2 regular expression syntax. The field label shows "RE2 regex" to remind you. The browser doesn't check the pattern before you save, so test it carefully against real completions.

Configuration examples

To set up the rule:

1. On the Rules page, select + New rule.
2. On the New rule page, enter format in Search templates, and then select Configure › on the Expected format absent card.

3. On the Configure rule page, for Project, choose the project this rule watches.
4. Check Name. It starts as the template name followed by the project slug. Rule names must be unique, so change it if you plan to create more than one rule from this template.
5. For Severity, select a level. The default is MEDIUM.
6. For Expected format, enter your pattern. To use one of the examples shown under the field, select it, and it's inserted for you.

7. Select Save & enable.

If you leave the pattern empty, the form shows "Expected format is required."

The following table shows the example patterns under the field.

Related

• Missing citation uses a pattern the same way, but checks for a citation marker.
• Banned term in output fails when the completion does match a pattern.
• Output length out of bounds checks the completion's length instead of its shape.
• Empty output catches blank completions, which this rule skips.

Next steps

• Create a rule
• Rules overview
• Explore traces