Data Freshness Policy

A Data Freshness Policy in ADOC ensures datasets remain current and available within the expected timeframes. Freshness is especially critical for real-time analytics, operational processes, and time-sensitive applications.

Unlike other monitoring policies, freshness alerts are not triggered by individual metrics. Instead, ADOC evaluates all configured rules together and generates alerts based on the overall policy score.

Example:

• Track if new sales data arrives every 24 hours.
• Detect if the asset size drops unexpectedly by more than 10%.
• Alert when row counts increase or decrease beyond expected limits.

Creating a Data Freshness Policy

You can create a Data Freshness Policy in two main ways:

Option 1: Through Manage Policies

1. Navigate to Data Reliability > Manage Policies.
2. Click Add Policy.
3. Select Freshness.
4. Choose the asset to monitor and click Next.
5. The Create Freshness Policy page opens for configuration.

Option 2: Through the Asset Details Page

1. Open the dataset in Asset Details.
2. Click the Actions button.
3. Select Data Freshness.
4. The Freshness & Volume Monitoring Settings page opens.

Policy Configuration Steps

Step 1: Enable Freshness Monitoring

1. Toggle Turn On Freshness & Volume Scoring.
2. Select which anomaly types to track:
   • Data Freshness: Alerts when updates do not arrive within the expected timeframe.
   • Absolute Row Count: Track row count against defined thresholds.
     • Change in Row Count: Detect row count increases/decreases.
     • Absolute Asset Size: Monitor asset size.
     • Change in Asset Size: Detect unexpected changes in size.

Each enabled anomaly creates a monitoring rule.

Step 2: Configure Thresholds

Thresholds define the expected freshness standards.

Policy Thresholds

• Success Threshold (0–100): Minimum score required for the policy to pass.
• Warning Threshold (0–100): Trigger a warning if the score falls below this percentage.
• Anomaly Strength Inclusion: Select High, Medium, or Low anomaly sensitivity.

Step 3: Alerts & Notifications (Optional)

Configure how ADOC notifies you when freshness issues are detected.

1. Severity Levels: Critical, High, Medium, Low.
2. Notification Channels: Email, Slack, Microsoft Teams, Webhook, ServiceNow, Chat.
3. Notify on Success: Optionally enable success notifications.
4. Re-notification Preferences:
   • Never: Only the first alert is sent.
   • After n failed runs: Alerts sent after repeated failures.
   • Every time: Alerts triggered for every failure.

Step 4: Advanced Freshness Settings (Optional)

Fine-tune model-based anomaly detection.

Step 5: Summary

1. Review selected anomalies and thresholds.
2. Confirm severity and notification settings.
3. Click Save Policy.

Executing a Data Freshness Policy

• Freshness Policies run automatically every hour.
• They cannot be manually executed.
• During execution, all configured rules are evaluated, and the final policy score is calculated.

Policy Scoring:

• Each rule contributes equally.
• A passing rule scores 100; a failing rule scores 0.
• The overall policy score is the average of all rule scores.

Alert Behavior:

• Alerts are based on the policy score, not on a single metric failure.
• If one rule fails (e.g., row count change), the overall score decreases, potentially triggering an alert.

Best Practices

• Continuous Alerting: Use Notify Every Time for real-time monitoring.
• Single Alert per Breach: Use Never for reduced noise until data is refreshed.
• Focused Alerts: Create separate policies if you want alerts for specific freshness thresholds only.