Schema Drift Policy

A Schema Drift Policy in ADOC ensures that datasets maintain a stable structure over time by detecting changes in table schemas or file schemas.

A policy is a set of rules that define the expected schema. A drift event occurs when a column is added, removed, or modified compared to the previously crawled schema.

Schema Drift policies run automatically every time a crawler is executed on the data source.

Example:

• A Customer Table Schema Drift Policy might include:
  • “Alert if new column customer_status is added.”
  • “Alert if data type of customer_id changes.”
  • “Alert if the email column is removed.”

Creating a Schema Drift Policy

You can create a Schema Drift Policy in two main ways:

Option 1: Through Manage Policies

1. Navigate to Data Reliability > Manage Policies.
2. Click Add New Policy (top-right).
3. Select Schema Drift as the policy type.
4. Choose the dataset (asset) to monitor.
5. The Create Schema Drift Policy page opens for configuration.

Option 2: Through the Asset Details Page

Overview Tab

1. Open the dataset in the Asset Details page.
2. In the Overview tab, click Actions > Add Schema Drift Policy.

Policies Tab

1. Navigate to the Policies tab.
2. Click Add Policy or use the Actions button and select Schema Drift.

Schema Configuration

Schema configuration defines which types of changes will trigger alerts.

Step 1: Select Drift Alerts

Enable one or more of the following toggles:

• Alert on Column Additions – Notifies when new columns appear.
• Alert on Column Removals – Notifies when columns are deleted.
• Alert on Column Metadata Changes – Notifies when metadata of existing columns changes.

Step 2: Edit Metadata (Optional)

Click Edit Metadata to select which metadata changes should be monitored.

• Standard Metadata: Data type changes.
• Relationships: Changes to foreign keys or references.
• Additional Metadata: Monitor advanced attributes such as column default, nullability, or interval type.

Click Done to save your selection.

Alerts & Notifications (Optional)

Define how and where alerts should be sent when schema drift is detected.

1. Severity Levels: Critical, High, Medium, Low.
2. Notification Channels: Email, Slack, Microsoft Teams, Webhook, ServiceNow, Chat.
3. Notify on Success: Optionally enable notifications for successful runs.
4. Re-notification Preferences:
   • Never: Suppress repeated alerts.
   • After n failed runs: Reduce noise by grouping violations.
   • Every time: Notify on each drift event.

Summary & Save

In the Summary section, review and confirm:

• Selected drift alerts and metadata options.
• Severity and notification channels.
• Success notification preference.

Click Save Policy to activate the Schema Drift Policy.

Executing a Schema Drift Policy

• A Schema Drift Policy cannot be run manually.
• It executes automatically each time the crawler runs on the dataset.
• When drift is detected, alerts are sent to configured channels.

Examples:

• Slack: Instant notification with details of schema changes.
• Email: Preview with a summary of added, removed, or modified columns.