Rules/RuleSets

On this page of ADOC, you can create rules to be included in the Data Quality policy that will apply to all qualifying assets. Ensure the same tag is assigned to the rule as is applied to all qualifying assets.

To create a rule, navigate to the Manage Policies page by selecting Reliability from the left pane and clicking the Rules/RuleSets tab, then click the Create Rule button.

Understanding the Rules Tables

The table on the Rules/RuleSets Page displays details of each rule created. Below are the descriptions of each column:

How to Create a Rule?

To create a rule, perform the following steps:

1. Click Create Rule.
2. Enter a name for the rule in the Name field.
3. (Optional) Enter a description for the rule in the Description field.
4. Select the rule type from the Rule Type drop-down menu.

Based on the rule type you select; you need to specify values for some of the additional fields. To learn more about Data Quality policy rules, see Data Quality Policy.

5. Add the tag to the rule. This tag must be the same tag which is added to all the qualifying assets. You can add multiple tags, if needed.
6. Select either All or Any in the Tag Match Condition field. This is applicable if you have added multiple tags. If you select the All option, only those assets which match all the tags are selected as qualifying assets. If you select the Any option, if an asset matches even a single tag, it is considered as a qualifying asset.
7. Click Save.

Create Rule Page

Rule Sets

A rule set is a group of rules. You must add the rules to a rule set to apply them on a data source.

You cannot apply a rule directly on a data source. To access the rule set page, click the Rule Sets tab on the Rules page.

The Rule Sets page displays the list of rule sets created in a table. The table has the following columns.

How to Import a Rule Set?

To import a rule set, perform the following steps:

1. Click the Import Rule Set button. The Import Rule Sets modal is displayed with the following steps to perform.

2. Click Start to drag and drop a file of either zip or gz file extension type or browse from your system files and click Upload.

3. Take action for rule sets found with same name on target assets. You can either Override the existing rule set with the one being imported or select Do not import. Note On selecting do not import none of the rule sets get imported.

4. Click Next. A successful import message is displayed. You must wait for the imports to happen.

How to Apply Rule Set?

To apply a rule set, follow the below steps:

1. Click the ellipsis button of the rule.
2. Select Apply Rule Set.

3. Select the source type to which the rule set must be applied in the Apply ruleset to field. This can either be a Data source, Database, Schema, or Big Query Dataset. While users can apply rule sets based on asset-level permissions, the actual creation of policies depends on domain-level policy permissions. If a user does not have the required permissions, policies may still be created, but they will not be visible to that user.
4. Select the name of the Data source, Database, Schema, or Big Query Dataset to which the rule set must be applied.

5. Click Apply.

How to Schedule a Rule Set?

1. Click the icon and select the Schedule Rule Set option. The Schedule Rule Set modal is displayed.
2. Provide a name for the schedule.
3. Select a group of assets to apply the selected rule set. For example, if you have selected data source, then select the data sources to apply the selected rule set to.
4. Enable the toggle button to automatically enable all created policies.
5. Select the time or day unit that you want to apply on the ruleset such as Minute, Hour, Day, Week, Month, or Year.
6. Click the Apply button to schedule to the rule set.

How to View Rules?

To view rules associated to the rule set, do the following:

1. Click the vertical ellipsis icon and select View Rules option. A modal is displayed with the following details:
   1. Name: Name of the rule.
   2. Applied By: The method in which the rule was applied. Example: Tag Based.
   3. Type: Type of rule applied. Example: Missing_Values, Distinctness_Check
   4. Created On: The date and time the rule was created.

How to View Schedules?

To view schedules of a rule set, do the following:

1. Click the vertical ellipsis icon and select View Schedules option. A modal is displayed with the following details:
   1. Name: Name of the schedule.
   2. Schedule: The schedule time.
   3. Enabled: A toggle displaying if the schedule is enabled or not.

Bulk Policies

Adding multiple data sources to ADOC can be time-consuming. Manually adding policies to assets can be cumbersome. ADOC's Bulk Policies feature simplifies this process by creating rules for data quality, grouping them, and applying them to data sources. ADOC automatically creates a Data Quality policy and applies the rules to assets matching a tag-based condition.

Terms Associated with Bulk Policies Feature

Before we proceed further, it is important to understand the following terms related to the Bulk Policies feature.

• Qualifying Assets: The qualifying assets are the assets on which quality policies need to be created and rules need to be added to all the policies. These assets would be referred to as qualifying assets through out this document.
• Bulk Policies: The data quality policy created on all the qualifying assets.
• Rule: The rule that must be added to all the quality policies which are created automatically, on the qualifying assets.
• Rule Set: A rule set is a group of rules. You must add a rule to a rule set to use this feature. Rule sets can now automatically propagate metadata and descriptions to their associated rules and policies.

How Bulk Policies Work?

The following steps outline how the Bulk Policies feature operates:

Tag Management: ADOC provides a Tags page where you can manage tags. You can apply several out-of-the-box ADOC tags to assets based on a regular expression match. You also have the option to create custom tags tailored to your organization’s requirements using regular expressions. Once these expressions apply a tag to the relevant assets, those assets qualify for bulk policies. For more information, see Tags.

Create Rules: After applying tags to the qualifying assets, you can define the rules that should be enforced on those assets. You must link rules to the same tag as tag management. Once you have created all the necessary rules, incorporate them into a rule set before applying it to a data source. For more information, see How to Create a Rule?

Create a Rule Set: You can apply a Rule Set to data sources, datasets, schemas, or Big Query datasets. All policies and rules generated from the rule set automatically apply the metadata and descriptions defined at the rule set level, ensuring full annotation of each policy and rule.

Apply Rule Set: You can apply the completed rule set to a specific data source. In addition to data sources, ADOC allows you to apply rule sets to datasets, schemas, or Big Query datasets. During this process, ADOC automatically includes the metadata and descriptions defined at the rule set and rule levels, ensuring that the policies and rules generated from the rule set are fully descriptive and traceable. For more information, see How to Apply Rule Set?

Rule Set Jobs: After applying the rule set to a data source, you can monitor the process in the Rule Set Jobs page. This page displays details such as the number of assets scanned, the number of policies created, and other relevant status information. Each job created is named after the rule set and tracks the rule set application across different data sources.