Policy APIs

This section provides comprehensive information on the APIs and methods available for managing policies in your data management workflow. From a data observability perspective, the Policy APIs serve as a foundation for maintaining data integrity, ensuring data quality, and meeting compliance requirements. By harnessing these APIs, organizations can proactively detect and resolve data issues, ensuring the reliable and accurate flow of data through their systems, ultimately contributing to improved data observability and trust in data-driven decision-making.

Retrieving a Policy Based on Its Identifier

This API allows you to find a specific policy using its identifier, which can be either the policy ID or policy name. Whether you have the ID or name, you can efficiently retrieve the desired policy. This simplicity streamlines policy access within your application.

Java
    
 
RuleResource policy = adocClient.getPolicy("153477");RuleResource policy = adocClient.getPolicy("redshift_automation_views_dq_null_check_01");
Copy

Retrieving a Policy Based on Type and Identifier

This API allows you to find a specific policy based on its type and identifier. The identifier can be either the policy ID or policy name, offering flexibility in searching for policies.

Java
    
 
RuleResource policy = adocClient.getPolicy(PolicyType.DATA_QUALITY, "153477");RuleResource policy = adocClient.getPolicy(PolicyType.DATA_QUALITY, "redshift_automation_views_dq_null_check_01");
Copy

Executing a Policy

The Acceldata SDK provides a utility function called executePolicy that allows you to execute policies both synchronously and asynchronously. When you use this function, it returns an object on which you can call getResult and getStatus to obtain the execution's result and status.

The required parameters for using the executePolicy function typically include the following:

Parameter	Description
policyType	This parameter is specified using an enum and is a required parameter. It accepts constant values as `PolicyType.DATA_QUALITY` or `PolicyType.RECONCILIATION`. It specifies the type of policy to be executed.
policyId	This is a required string parameter. It specifies the unique identifier (ID) of the policy that needs to be executed.
optionalSync	This is an optional Boolean parameter. It determines whether the policy should be executed synchronously or asynchronously. The default value is `false`. If set to `true`, the execution will return only after it has completed. If set to `false`, it will return immediately after the execution begins.
optionalIncremental	This is an optional enum parameter used to determine the behavior in the event of a failure. The default value is `FailureStrategy.DoNotFail`. The FailureStrategy enum can have three values: `DoNotFail`: It will never throw an exception. In the event of a failure, the error will be logged. `FailOnError`: It will only throw an exception if there is an error. In the event of a warning, it will return with no errors. `FailOnWarning`: It will throw exceptions on both warnings and errors.
pipelineRunId	This is an optional parameter specified as a long. It represents the run ID of the pipeline where the policy is being executed. This parameter can be used to link the execution of the policy with a specific pipeline run, providing context for the execution.

Java
    
​x
 
// Specify the policy type PolicyType policyType = PolicyType.RECONCILIATION;​// Specify the unique identifier (ID) of the policy to be executedlong policyId = 154636L;​// Set the execution mode to synchronous (wait for completion)boolean optionalSync = true;​// Specify that the policy execution should be full, not incrementalboolean optionalIncremental = false;​// Set the failure strategy to FailOnError (throw exception on error)FailureStrategy optionalFailureStrategy = FailureStrategy.FailOnError;​// You can optionally specify a pipeline run ID if neededOptional<Long> pipelineRunId = Optional.empty();​// Execute the policy with the specified parametersExecutor executor = adocClient.executePolicy(    policyType,    policyId,    Optional.of(optionalSync),    Optional.of(optionalIncremental),    Optional.of(optionalFailureStrategy),    pipelineRunId);
Copy

Getting Policy Execution Result

To access the outcome of policy execution, you have two options. You can call adocClient.getPolicyExecutionResult method directly on the adocClient , or you can opt to use the getResult method provided by the executor object generated during the policy execution. Also in case the policy execution is in a non terminal state, it will be retried till the execution reaches a terminal state. The getPolicyExecutionResult parameters include:

Parameter	Descritption
policyType	This parameter is mandatory and is specified using an enum, which is like a set of predefined options. It tells the system what kind of policy to apply. It can take two values: `PolicyType.DATA_QUALITY` or `PolicyType.RECONCILIATION`.
executionId	This is a required parameter, and it's a string value. It represents the unique identifier of the execution you want to query for results. You need to specify this identifier to retrieve specific results.
optionalFailureStrategy	This parameter is optional, and it's also an enum parameter. It helps decide what to do in case something goes wrong during the execution. The default behavior, if you don't specify this, is to not fail (`FailureStrategy.DoNotFail`). This means that if there is a problem, it will be logged, but the execution continues.

Java
    
 
// Specify the failure strategy as "FailOnError"Optional<FailureStrategy> optionalFailureStrategy = Optional.of(FailureStrategy.FailOnError);​// Set sleep interval to 15 (optional)Optional<Integer> sleepInterval = Optional.of(15);​// Set the retry count to 5 (optional)Optional<Integer> retryCount = Optional.of(5);​// Call getResult on the executor object with optional parametersRuleResult ruleResult = executor.getResult(    sleepInterval,    retryCount,    optionalFailureStrategy);​// Alternatively, you can call getPolicyExecutionResult on the adocClient// to retrieve the policy execution resultRuleResult ruleResult = adocClient.getPolicyExecutionResult(    PolicyType.DATA_QUALITY,    executor.getId(),    optionalFailureStrategy);
Copy

Retrieving Current Policy Execution Status

To find out the current status of a policy execution, you have two options. You can either call the getPolicyStatus method on the adocClient, or you can call the getStatus method on the executor object. These methods will provide you with the current resultStatus of the execution, which gives information about where the execution stands at the moment.

The required parameters for getPolicyStatusinclude:

Parameter	Description
policyType	The policy type is specified using an enum parameter. It is a required parameter. It is an enum that will accept constant values as `PolicyType.DATA_QUALITY` or `PolicyType.RECONCILIATION`
executionId	The execution id to be queried for the result is specified as a string parameter. It is a required parameter.

Java
    
//Using Executor object returned during executePolicy callPolicyExecutionStatus policyExecutionStatus = executor.getStatus();​//or adocClient getStatus methodPolicyExecutionStatus policyExecutionStatus = adocClient.getPolicyExecutionStatus(PolicyType.DATA_QUALITY,executor.getId());
Copy

Executing Policies Asynchronously

Here's an example that demonstrates how to initiate policy execution in an asynchronous manner.

Java
    
// Define connection details and create an AdocClientString baseUrl = "baseUrl";String accessKey = "accessKey";String secretKey = "secretKey";​// Create an AdocClient with the specified connection detailsAdocClient adocClient = AdocClient.create(baseUrl, accessKey, secretKey).build();​Long policyId = 88888L;​// Set optional parameters for asynchronous policy executionOptional<Boolean> optionalSync = Optional.of(false); // Set sync flag to false for async executionOptional<Boolean> optionalIncremental = Optional.of(false); // Optional: set execution mode to non-incrementalOptional<FailureStrategy> optionalFailureStrategy = Optional.of(FailureStrategy.FailOnError); // Optional: define failure strategyOptional<Long> optionalPipelineRunId = Optional.empty(); // Optional: specify a pipeline run ID​// Get an asynchronous policy executorExecutor asyncExecutor = adocClient.executePolicy(    PolicyType.DATA_QUALITY,    policyId,    optionalSync,    optionalIncremental,    optionalFailureStrategy,    optionalPipelineRunId);​// Wait for the execution to complete and get the final resultRuleResult ruleResult = asyncExecutor.getResult(    optionalSleepInterval, // Optional: sleep interval    optionalRetryCount, // Optional: retry count    optionalFailureStrategy // Optional: failure strategy);​// Get the current status of the asynchronous policy executionPolicyExecutionStatus policyExecutionStatus = asyncExecutor.getStatus();
Copy

Executing Policies Synchronously

Here's an example that demonstrates how to initiate a synchronous policy execution, where the execution waits for completion before proceeding:

Java
    
// Define connection details and create an AdocClientString baseUrl = "baseUrl";String accessKey = "accessKey";String secretKey = "secretKey";​// Create an AdocClient with the specified connection detailsAdocClient adocClient = AdocClient.create(baseUrl, accessKey, secretKey).build();​Long policyId = 88888L;​//Set optional parameters for asynchronous policy executionOptional<Boolean> optionalSync = Optional.of(true); // Set sync flag to false for async executionOptional<Boolean> optionalIncremental = Optional.of(false); // Optional: set execution mode to non-incrementalOptional<FailureStrategy> optionalFailureStrategy = Optional.of(FailureStrategy.FailOnError); // Optional: define failure strategyOptional<Long> optionalPipelineRunId = Optional.empty(); // Optional: specify a pipeline run ID​// Get an asynchronous policy executorExecutor syncExecutor = adocClient.executePolicy(    PolicyType.DATA_QUALITY,    policyId,    optionalSync,    optionalIncremental,    optionalFailureStrategy,    optionalPipelineRunId);​// Wait for the execution to complete and get the final resultRuleResult ruleResult = syncExecutor.getResult(    optionalSleepInterval, // Optional: sleep interval    optionalRetryCount, // Optional: retry count    optionalFailureStrategy // Optional: failure strategy);​// Get the current status of the synchronous policy executionPolicyExecutionStatus policyExecutionStatus = syncExecutor.getStatus();
Copy

Enabling a Policy

The provided code demonstrates how to enable a rule or policy.

Java
    
 
String dqPolicyNameToEnableDisable = "sdk_dq_policy_to_enable_disable";RuleResource dqRule = adocClient.getDataQualityRule(dqPolicyNameToEnableDisable);Long policyId = dqRule.getRule().getId();GenericRule rule = adocClient.enableRule(policyId);
Copy

Disabling a Policy

The provided code demonstrates how to disable a rule or policy.

Java
    
 
String dqPolicyNameToEnableDisable = "sdk_dq_policy_to_enable_disable";RuleResource dqRule = adocClient.getDataQualityRule(dqPolicyNameToEnableDisable);Long policyId = dqRule.getRule().getId();GenericRule rule = adocClient.disableRule(policyId);
Copy

Canceling Policy Execution

The provided code demonstrates how to cancel a policy execution using both the executor object and the adocClient.

Using the executor object:

Java
    
 
// Execute a data quality policy asynchronouslyExecutor asyncExecutor = adocClient.executePolicy(    PolicyType.DATA_QUALITY,    153477L,    Optional.of(false),    Optional.of(false),    Optional.of(FailureStrategy.FailOnError),    Optional.empty());​// Cancel the execution using the executor objectCancelRuleResponse cancelRuleResponse = asyncExecutor.cancel();
Copy

Using adocClient:

Java
    
 
// Cancel a specific rule execution using the adocClientadocClient.cancelRuleExecution(153477L);
Copy

Listing All Policies

Retrieving a list of all policies from your catalog server while applying specified filters.

Parameter	Description
policyFilter	PolicyFilter can have the following parameters `period`: Allows you to filter policies based on a specified time period. It's defined as an enum called `ExecutionPeriod` . `tags`: Enables filtering by tags, allowing you to narrow down policies that have specific tags associated with them. `lastExecutionResult`: Permits filtering based on the results of the last execution, helping you identify policies with particular execution outcomes. `assets`: This parameter is used to filter policies associated with specific assets. It expects a list of objects of type `Asset` returned as a filter criterion. `dataSources`: You can filter policies by specifying particular datasources. This parameter also requires a list of objects of type `Datasource`. `policyType`: This parameter filters policies by their type and is defined as an enum of type `PolicyType`. `enable`: By setting `enable` to `true`, you can filter for policies that are currently enabled. `active`: When `active` is set to `true`, you filter for policies that are currently active.
optionalPage	Specifies the page number of the query output, allowing you to navigate through paginated results.
optionalSize	Defines the size of each page, controlling how many policies are displayed on each page of the query output.
optionalWithLatestExecution	This option allows you to include the latest execution details of the policy in the query output. When enabled, it fetches information about the most recent execution of each policy.
optionalSortBy	This parameter determines the sorting order of the policies in the query output. You can specify the order in which policies are presented, for example, by name or execution date.

Java
    
 
// Define filter criteria using variablesPolicyType policyTypeFilter = PolicyType.DATA_QUALITY; // Filter by policy type (e.g., DATA_QUALITY)boolean activeFilter = true; // Filter to include only active policiesboolean enableFilter = true; // Filter to include only enabled policiesOptional<Integer> pageNumber = Optional.empty(); // Optional: Page number of the query outputOptional<Integer> pageSize = Optional.empty(); // Optional: Size of each pageboolean fetchLatestExecution = false; // Optional: To fetch the latest execution of the policy (set to false)Optional<String> sortingOrder = Optional.empty(); // Optional: Sorting order​// Create a PolicyFilter using the defined variablesPolicyFilter policyFilter = PolicyFilter.builder()    .policyType(policyTypeFilter)    .active(activeFilter)    .enable(enableFilter)    .build();​// List all policies based on the provided filter and optional parametersList<? extends Rule> allPolicies = adocClient.listAllPolicies(    policyFilter,    pageNumber,    pageSize,    Optional.of(fetchLatestExecution),    sortingOrder);
Copy

Last updated on

Was this page helpful?