Data Freshness Policy

A Data Freshness policy checks whether an asset is updated on time and whether its size and row count change as expected. Use it to catch late, missing, or unusual data loads before they affect reports and downstream pipelines.

ADOC reads metadata from the data source, such as the last update time, the asset size, and the row count. It compares these values with thresholds that you set, and it can also detect anomalies by learning the asset's usual update pattern.

When to use a Data Freshness policy

Use a Data Freshness policy when you need to:

• Confirm that daily sales data loads before morning business reviews.
• Check that real-time sensor data isn't delayed.
• Verify that hourly customer activity feeds arrive on schedule.
• Detect upstream pipelines that fail without reporting an error.
• Get alerts when data isn't updated within an expected interval.

How Data Freshness policies work

Each policy monitors one asset and can track the following metrics:

You can monitor each metric in two ways, and you can use both on the same metric:

• Thresholds: You define the expected values, such as how often the asset must update or the allowed row count range. The run fails when a value is outside these limits.
• Anomaly detection: ADOC learns the metric's usual pattern from earlier runs and flags values that are unusual, even when they're within your thresholds.

ADOC combines the results into a score for each run and compares it with the policy's Success Threshold and Warning Threshold. For more information, see How the policy score is calculated.

Create a Data Freshness policy

You can start from Manage Policies or from Discover Assets.

Start from Manage Policies

1. In the left navigation menu, go to Data Observability Cloud > Governance > Policies.
2. Select Add Policy, and then select Freshness.
3. In Select Asset to Create Policy, select the asset, and then select Next.

Start from Discover Assets

1. Go to Data Observability Cloud > Discover > Assets.
2. In the asset list, find the asset, and then select it to open the Asset Details page.
3. Click the Actions button, and select Data Freshness.

The policy page opens with five configuration steps and a summary. The following sections describe each step.

Step 1: Enable freshness

1. Turn on the Turn On Freshness & Volume Scoring toggle.
2. Under Anomaly Types, turn on anomaly detection for each metric that ADOC should check for unusual behavior: Data Freshness, Change in Asset Size, Absolute Asset Size, Change in Row Count, or Absolute Row Count.
3. Select Configure Thresholds.

Anomaly types control anomaly detection only. You can still set thresholds for a metric in step 2 even if you don't turn on its anomaly type.

Step 2: Configure thresholds

In Metrics, select a metric, configure its rule, and then select Add Rule. Repeat for each metric that you want to check against fixed limits.

Data Freshness: Uses a Time Based rule. In Look Back Window, enter a number, and in Look Back Window Type, select Days, Hours, or Minutes. For example, 24 Hours means that ADOC reports a failure when the asset hasn't updated within the last 24 hours.

Size and row count metrics: Change in Asset Size, Absolute Asset Size, Change in Row Count, and Absolute Row Count each offer two threshold types:

• Absolute: Select a Direction of Above, Below, or Range, and then enter the success limits and, optionally, the warning limits. Size metrics use bytes. A band below the fields shows when the rule passes or fails.
• Relative: Compares the current value with its recent history. Select a Comparison Operator of Increased By, Decreased By, or Changes By, enter Success Threshold % and, optionally, Warning Threshold %, and set the Look Back Window and Look Back Window Type.

Then set the policy-level settings:

• Success Threshold (0-100): The minimum score for the policy to pass.
• Warning Threshold (Optional: 0-100): A lower limit. Scores between the warning threshold and the success threshold show a Warning.
• Only include anomalies in scoring when the anomaly strength is greater than or equal to: Select HIGH, MEDIUM, or LOW. Anomalies below this strength don't reduce the score.

Step 3: Alerts and notifications (optional)

• Alerts: Select the alert severity: Critical, High, Medium, or Low.
• Notifications: Select one or more notification channels.
• Notify On Success and Notify On Warning: Select to also receive notifications when a run passes or shows a warning.
• When do you want to be Re-Notified?: Applies while the policy stays in violation. Select Reduce Noise: Never, Reduce Noise: Send notification every failed policy run, or Notify Every Time.

Step 4: Advanced freshness settings (optional)

These settings control anomaly detection:

• Training Window Minimum: The number of days of history that ADOC collects before it starts detecting anomalies. Enter at least 3 days. ADOC also needs at least three executions.
• Model Sensitivity: Select HIGH, MEDIUM, or LOW. Higher sensitivity flags smaller deviations.

Step 5: Execution details (optional)

By default, you run the policy manually. To run it on a schedule:

1. Turn on Schedule policy execution.
2. Select a Time zone.
3. In Schedules, select Add. Select Hourly, Daily, Weekly, Monthly, or Custom (a cron expression), and set the time.
4. Select Add Schedule, and then select Save. To run the policy at more than one time, select Add another time.

ADOC shows the next execution time under the schedules.

Step 6: Summary

1. Review the Anomaly Detection and Thresholds table and the other settings.
2. Optionally, add tags.
3. Select Save Policy.

How the policy score is calculated

Each metric can contribute up to two evaluations to a run: one for its threshold rule and one for anomaly detection, if you turned it on. ADOC calculates the score as follows:

Score = ((Total evaluations − Failed evaluations) ÷ Total evaluations) × 100

Example: You set a 24-hour Data Freshness rule and turn on Data Freshness anomaly detection, so each run has two evaluations.

• If the data arrives on time with no anomaly, no evaluations fail, and the score is 100.
• If the data arrives late but within 24 hours, anomaly detection flags it, one evaluation fails, and the score is 50.

With a Success Threshold of 100 and a Warning Threshold of 50, the second run shows a Warning.

Run a policy and view results

To run a policy on demand, open the policy and select Run.

To view results:

1. Go to the Manage Policies page, and select the Freshness filter.
2. Select the policy name to open its execution history.

Each execution shows:

• Overall summary: The total number of evaluations and how many passed or failed.
• Evaluation metrics: Each configured metric, with its threshold and anomaly results.
• Expected range and actual value: The allowed range for each metric and the value that ADOC observed.
• Anomalous or SLA status: Whether the metric breached a threshold or was flagged as an anomaly.
• Strength: How confident ADOC is in a detected anomaly.
• Feedback: Select thumbs up or thumbs down to help improve anomaly detection.

A trend chart below the summary shows how each metric changes across runs, so you can spot late arrivals and unusual spikes.

Example: Catch a stale sales dashboard

An executive dashboard shows the previous day's sales every morning at 8 AM. The overnight ETL job sometimes fails without an error, and nobody notices until after the dashboard is used. To catch the problem in time:

1. Create a Data Freshness policy on the sales table.
2. Turn on Data Freshness anomaly detection.
3. Add a Data Freshness rule with a Look Back Window of 8 Hours.
4. Schedule the policy to run daily at 6:30 AM, and send Critical alerts to the data team's channel.

If the load is late, the team has time to fix it before 8 AM.

The following look back windows are common starting points:

Supported data sources

Data Freshness policies use the Pushdown engine, which reads metadata directly from the source, for the following data sources:

• Snowflake
• BigQuery
• Redshift
• Databricks
• PostgreSQL
• MySQL
• Oracle
• Azure Synapse
• SingleStore (MemSQL)

For Hive and file-based data sources, ADOC uses the Spark engine. For more information, see Pushdown Data Engine.

Limitations

• The first run sets the baseline. Change metrics are calculated from the second run onward.
• Size units, such as bytes, KB, or MB, depend on the data source.
• Anomaly detection starts only after the Training Window Minimum and at least three executions.
• ADOC detects and reports delays. It doesn't fix them.

Best practices

• Base thresholds on when people need the data: Add a buffer for normal variation and troubleshooting time.
• Start with a longer look back window: Begin at about twice the expected delay, and shorten it as you learn the pattern.
• Use anomaly detection for data with variable timing: It catches gradual slowdowns that a fixed threshold misses.
• Match sensitivity to the pattern: Use MEDIUM for regular daily loads, LOW for variable timing, and HIGH for critical real-time data.
• Monitor the whole chain: Add policies to source, intermediate, and final tables to find where a delay starts.
• Review thresholds after pipeline changes: A new schedule or volume can make old limits wrong.

Common mistakes to avoid

• Setting the look back window equal to the expected arrival time, with no buffer.
• Using HIGH sensitivity on data that arrives at irregular times, which causes false alerts.
• Forgetting weekends and holidays when you set daily expectations.
• Alerting on an asset without checking whether its upstream sources are late.

Troubleshooting

The alert says data is stale, but it looks current:

• Check that the policy's last run succeeded.
• Confirm that the time zone in the schedule matches your data.

You get too many false alerts:

• Increase the look back window, or lower Model Sensitivity.
• Raise the minimum anomaly strength included in scoring.
• Check that the Training Window Minimum covers enough history.

You don't get alerts when data is stale:

• Confirm that the policy has a schedule, or run it manually.
• Confirm that Turn On Freshness & Volume Scoring is on.
• Check that notification channels are selected.