Data Anomaly Policy

A Data Anomaly Policy in ADOC detects unusual patterns or irregularities in data that may indicate data quality issues, system faults, or significant business events. By leveraging anomaly detection across structured and semi-structured data types (Array, Object, Variant in Snowflake), ADOC ensures robust, versatile monitoring across modern data formats.

Use case examples:

• Detect sudden drops in transactions.
• Spot unexpected spikes in data volume.
• Identify irregular values in semi-structured data like JSON logs.

Creating a Data Anomaly Policy

Create a Data Anomaly Policy from:

Option 1: Through Manage Policies

1. Navigate to Data Reliability > Manage Policies.
2. Click Add Policy (top-right).
3. Select Data Drift as the policy type.
4. Choose the dataset (asset) to monitor.
5. The Create Data Drift Policy page opens for configuration.

Option 2: Through the Asset Details Page

Overview Tab

1. Open the dataset in the Asset Details page.
2. In the Overview tab, click Actions > Add Data Drift Policy.

Policies Tab

1. Navigate to the Policies tab.
2. Click Add Policy or use the Actions button and select Data Drift.

Policy Configuration Steps

Step 1: Policy Details

1. Processing Engine: Choose either Spark or Pushdown as the processing engine.
2. Click Show Columns to view the dataset schema.

Step 2: Select Data Columns

By default, all columns are included for profiling. To enable anomaly detection for specific columns:

1. Check the box under the Data Anomaly column.
2. Click Configure Anomaly to continue.

Step 3: Configure Anomaly Rules

Policy Scoring

1. Policy Thresholds:

   • Success Threshold (0–100): Minimum score for passing.
   • Warning Threshold (0–100): Trigger a warning if the score drops below this value.

2. Anomaly Strength Inclusion:

   • Select the minimum anomaly strength to include in scoring (High, Medium, Low).
   • Example: Only include anomalies with strength >= HIGH.

Alerts & Notifications

Define how ADOC notifies you when anomalies are detected.

1. Severity Levels: Critical, High, Medium, Low.
2. Notification Channels: Email, Slack, MS Teams, ServiceNow, Webhook, Chat.
3. Notify on Success: Optionally enable success alerts.
4. Re-notification Preferences:
5. Reduce Noise (Never): One-time alerts only.
6. Send Every [X] Failed Runs: Alerts after repeated violations.
7. Notify Every Time: Alerts triggered for every violation.

Anomaly Detection Settings

Fine-tune anomaly detection behavior:

Advanced Performance Settings

Optimize profiling for large or wide datasets.

1. Processing Batch Size:

   • Default: 30
   • Adjust only under support guidance.

2. Optimize Data Load with Batches:

   • Loads data in smaller chunks instead of all at once.
   • Useful for wide tables with frequent updates.
   • May introduce slight inconsistencies in results.

Step 4: Summary

1. Review policy details, thresholds, alerts, and performance settings.
2. Click Save Policy to finalize.

Executing Data Anomaly Policies

Once created, anomaly policies are executed during asset profiling.

Method 1: From Manage Policies > Profiles (Beta)

1. Go to Profiles.
2. Search for the asset with the policy applied.
3. In the Profiles table, click Play.
4. Profiling begins and the Data Anomaly Policy executes automatically.

Method 2: From Asset Details Page

1. Open the Asset Details page.
2. Click Actions.
3. Select profiling type (Full or Selective).
4. Profiling triggers the anomaly policy.