User Defined Templates
User Defined Templates let you define a User Defined Function (UDF) once and use the same UDF definition across various other Data Quality policies.
A program or environment user can provide a user-defined function to organize their code within the body of policy.
Once you have defined a function, you can use it just like the built-in action and parser functions.
Functions receive variables by reference rather than by value
User-Defined Types (UDTs) on Different Spark Runtimes:
- Spark on Kubernetes: All types of UDTs are supported, including SQL, Scala, Java, Python, and JavaScript.
- Spark on Other Runtimes: Runtimes such as Databricks, Dataproc, and HDFS only support SQL- and Scala-driven UDTs.
To use the User Defined Templates:
- Click the
Settings icon. - Click User Defined Templates. The user-defined templates page is displayed.
ADOC allows you to create the following two types of user-defined templates:
ADOC recommends you use Scala or Java languages for UDT template creation. Python and JavaScript have issues with huge volumes of data.
The Transform User Defined Template allows you to extract values from a record in a data asset (table).
To create a Transform UDT, perform the following steps:
- In the main menu navigation, click on the Settings icon.
- Click on user-defined templates. A pop-up dialogue box is displayed.
- Click Transform UDT, and then click Continue. The Define Transform UDT panel appears.
- Enter the following parameters:
| Parameter | Definition |
|---|---|
| Name | Provide a unique name for the Transform UDT. |
| Description | Provide a description to easily identify the purpose of the Transform UDT. |
| Language | Select one of the following languages that are listed in the Language drop-down list: Python, Java, Scala, and JavaScript. You can also select Expression. this allows you to enter an expression directly. |
| UDF Data Type | A separate list of data types is provided depending on the programming language chosen. Choose a datatype for the UDF. |
| Code | Enter your code using the correct syntax for the programming language you have chosen. |
| Reference Variables | If your code consists of variables, they are automatically added under the Reference Variables section.
|
Click the Validate drop-down button and select one of the following ways that you would like to validate your code:
Click Save. The new Transform UDF template is saved.
Validation User Defined Template allows you to filter a record in a data asset (table).
To create a Transform UDT, perform the following steps:
- Click Settings.
- Click User Defined Templates. A pop-up dialogue box is displayed.
- Click Validation UDT and click Continue. The Define Validation UDT panel is displayed.
- Enter the following parameters:
| Parameter | Definition |
|---|---|
| Name | Provide a unique name for the Validation UDT. |
| Description | Provide a description to easily identify the purpose of the Validation UDT. |
| Language | Select one of the following languages that are listed in the Language drop-down list: Python, Java, Scala, and JavaScript. You can also select Expression. This allows you to enter an expression directly. |
| Code | Enter your code using the correct syntax for the programming language you have chosen. |
| Reference Variables | If your code consists of variables, they are automatically added under the Reference Variables section.
|
| Labels | Attach Labels (key value pair) to your UDF templates. |
| Lookup Type | Enable this toggle switch to generate Lookup type rule template. |
| With Metadata | Enable this toggle switch to add metadata for the various checks you want to embed and receive feedback in addition to the standard Boolean results. For more information, on how to create using the CI/CD see Interfaces. |
From V2.7.0 onwards the Metadata toggle switch has been added to the user interface.
With Metadata : Error type displayed
Click the Validate drop-down button and select one of the following ways that you would like to validate your code:
Click Save. The new Validation UDF template is saved.
Validation Methods
You can validate the code provided when creating any type of user-defined template, either manually or with existing assets in a data source.
Manual Validation
To perform a manual validation, follow the steps below:
- Click Manual from the Validate button drop-down list. The Enter Test Data pop-up is displayed.
- Enter your data in the space provided.
- Select an analytics pipeline to run the provided data.
- Enter Values for all of the reference variables used in your code.
- Click the Validate button.
Asset Validation
To validate your code using assets from within your data source, follow these steps:
- From the Validate button drop-down list, select With Assets. The Select Asset pop-up window appears.
- Use the search bar to find an asset by its name, or navigate directly through the required data source and select an asset.
- Click Next. The Set Reference Variables pop-up windows is displayed.
- Fill in the values for all of the reference variables used in your code.
- Click the Validate button.
Actions Available for User-Defined Templates
After successfully saving a user-defined template, it appears on the User Defined Templates (UDT) page. You can perform the following actions on each UDT by selecting the
Edit: Opens the Edit User-Defined Template page, where you can modify the template. This page also displays the linked policies that may be impacted by the changes. View: Displays the details of the UDT, including its name, creator, last updated by, language, and code.
Linked Policies: Shows a list of all policies linked to the UDT.
Delete: Deletes the selected UDT. A confirmation dialog appears before the deletion is finalized.