Persistence Configuration

Persistence Configuration defines where and how ADOC stores policy execution outputs, including:

Good records
Bad records
Summary (Metadata) records

Starting from ADOC 26.3.0, persistence management is centralized and template-driven. This allows organizations to:

Standardize storage paths across teams
Organize outputs by policy, datasource, or time
Improve governance and auditability

Persistence behavior is controlled at two levels:

Tenant Level: Define reusable storage configurations
Policy Level: Choose or override the tenant configuration

The storage base path and credentials continue to be defined at the data plane level.

How persistence works in ADOC

Persistence path resolution happens in layers:

Data plane - Global Storage Secret

Defines:

Storage type
Type specific information like bucket / container
Credentials

This remains unchanged in 26.3.0.

Dataplane - Application Configuration

Defines:

MEASURE_RESULT_SAVE_PATH
- Optional path component used for result storage.
- For LOCAL storage types: Treated as the full base path (supports volume mounts).
- For Non-LOCAL storage types (e.g., S3, ADLS, GCS): Appended to the base bucket/container path.

Example (Non-LOCAL): If base path = s3a://bucket and MEASURE_RESULT_SAVE_PATH = dq-results

Final base becomes: s3a://bucket/dq-results

Tenant-Level persistence configuration

Defines:

Path suffix templates
File formats for Good and Bad records
Default behavior for policies

Policy-Level persistence configuration

Allows a policy to:

Use the tenant default configuration
Select a saved configuration
Define a custom path template

Write to multiple storage buckets

A single data plane can direct policy results to different storage buckets by setting a fully qualified storage URI as the Base Path in a Persistence Configuration. This allows organizations to separate results by compliance boundary, application domain, or infrastructure team ownership without provisioning a separate data plane per bucket.

How base path resolution works

When a policy executes, the platform resolves the storage destination for each result category as follows:

Base Path Value	Result
Fully qualified URI — for example `s3://my-bucket/results/` or `gs://my-bucket/results/`	The specified bucket is used directly, overriding the Dataplane's global storage setting.
Relative path — for example `domain-a/finance/results`	The path is appended under the Dataplane's global base path. The bucket does not change.
Not set (no custom configuration assigned)	The Dataplane's default global storage path is used. Existing behavior, unchanged.

Access credentials (access key, secret key, instance profile, or equivalent) are always sourced from the Dataplane's global storage configuration. Credentials are not overridden per policy or per bucket.

Supported storage types

Amazon S3 — fully qualified URI using the s3:// or s3a:// scheme.
Google Cloud Storage — fully qualified URI using the gs:// scheme.
ADLS — per-policy path overrides append to the global base path only. Container-level overrides for ADLS are not supported in this release.

How to use persistence configuration

Required permissions

The following permissions are required to manage Persistence Configurations:

Permission	Required for
`DATA_PERSISTENCE_VIEW`	Viewing existing persistence configurations.
`DATA_PERSISTENCE_CREATE`	Creating new persistence configurations.
`DATA_PERSISTENCE_MODIFY`	Updating or deleting existing configurations.

Configure Tenant-Level persistence

If you want consistent storage structure across policies, configure it at the tenant level.

Navigate to Settings -> Data Reliability -> Persistence Configurations. You will see a list of existing configurations.

Note Every tenant automatically has a default configuration named: Acceldata

For existing tenants, this default preserves backward-compatible paths.
For new tenants, it is created during onboarding.

Note Any saved tenant-level configuration can be designated as the new default configuration for the tenant. You may use this default as-is or create new configurations.

Create a new configuration

Click the Add Configuration button on the Persistence Configuration page.

Each configuration defines:

Good Records Template
Bad Records Template
Metadata Template
Output Format (Good/Bad only)

Note

Summary (Metadata) records are always stored in JSON format.
Templates define only the suffix.
Full paths such as: s3://bucket/… are not allowed in templates.
The final storage path is: <Dataplane Base Path> + <Dataplane Save Path (if configured)> + <Template Suffix>

Example Template

{{DATASOURCE_NAME}}/{{POLICY_NAME}}/{{REQUEST_ID}}/{{RECORD_TYPE}}

This would organize results by:

Datasource
Policy
Execution
Record type

Apply configuration at the policy level

When editing or creating a policy by navigating to Manage Policies → Open Policy → Edit → Advanced Settings, under persistence configuration, click the drop-down to find three options:

SYSTEM DEFAULT Uses tenant default configuration at execution time. If tenant default changes, future executions of that policy automatically use the new default.
Saved Configuration This configuration explicitly binds the policy to a selected configuration. Even if this configuration is also the tenant default, it is locked to the policy. Note that changes to tenant default do NOT affect this policy. Use this when:
- Different teams require different folder structures.
- Certain policies must always follow a specific storage layout.
CUSTOM Allows you to define custom suffix templates directly in the policy. Use this when:
- Only one policy needs a unique structure.
- You want to temporarily override the tenant configuration.

All validation rules apply to custom templates. Define custom suffix templates directly in the policy.

Change good/bad record path for a policy

Go to Manage Policies.
Open a policy -> View Policy -> Edit Policy. The Edit Policy page is displayed with the policy summary details.
Go to the Configure Execution Details step and click the Advanced Execution Settings dropdown.
Under Persistence Configuration, select one of the following: System Default, Active configurations, or CUSTOM.
Define Good records, Bad records, and Metadata templates.
Click Save.

From the next execution onward, records will be written to the new path.

Note Previous executions prior to changing the Good/Bad record path for this policy remain unchanged.

Manage persistence configurations using APIs

In addition to configuring persistence paths through the ADOC user interface, administrators and automation workflows can also manage persistence configurations using REST APIs.

These APIs allow you to:

Retrieve the persistence configuration associated with a policy
Update policy-level persistence settings
Create and manage tenant-level persistence configurations
Validate and preview suffix templates
Retrieve available template variables
Determine whether a dataplane supports templated persistence paths

This enables organizations to automate persistence configuration management as part of CI/CD pipelines, governance workflows, or infrastructure automation.

For detailed API reference and examples, see Persistence Configuration APIs.

Template variables

Templates use predefined variables to dynamically build storage paths. Click View Available variables to view the Available Template Variables wizard on the right.

Variable	Description
`{{POLICY_TYPE}}`	Policy type, lower cased (data-quality, reconciliation, auto-anomaly)
`{{POLICY_NAME}}`	Policy name (sanitized for path safety)
`{{POLICY_ID}}`	System policy ID
`{{REQUEST_ID}}`	Unique ID per execution (recommended for uniqueness)
`{{EXECUTION_ID}}`	Internal execution ID
`{{RECORD_TYPE}}`	successrecords, errorrecords, summary, topanomalies
`{{ASSET_NAME}}`	Asset name (sanitized)
`{{DATASOURCE_NAME}}`	Datasource name (sanitized)
`{{TIME:format}}`	Execution start time in Joda-Time format
`{{EPOCH_MILLIS}}`	Execution timestamp in milliseconds

Important template rules

Must include {{RECORD_TYPE}} to ensure that Good, Bad, and Metadata records are stored in separate folders.
Must include atleast one {{REQUEST_ID}} or{{EXECUTION_ID}}to prevent overwriting results from multiple executions.
Allowed Characters in Static Text:
- Unicode Letters
- Unicode Numbers
- _ (underscore)
- '-' (hyphen)
- = (equals)
Use / to separate path segments.
Run Time Sanitization: User-defined values are sanitized automatically. User-defined values such as:
- POLICY_NAME
- ASSET_NAME
- DATASOURCE_NAME

are automatically sanitized.

The format defined will only be used by spark executions. Pushdown executions use the format set in datasource configuration.

Sanitization behavior:

/ replaced with __
Reserved names (e.g., CON) wrapped as __CON__
Invalid characters replaced
Leading/trailing dots removed
Maximum length: 200 characters

Common path suffix patterns

Organize results by team

To store execution results in team-specific folders, create a tenant-level persistence configuration with a static team folder name.

Example: TEAM_A/{{POLICY_NAME}}/{{EXECUTION_ID}}/{{RECORD_TYPE}}

After you save the configuration, select it in the policy settings.

All executions of that policy will write results under the TEAM_A folder.

Use this approach when multiple teams share the same storage location but need logical separation.

Organize results by data source

To group results by datasource, use the {{DATASOURCE_NAME}} variable in the template.

Example template: {{DATASOURCE_NAME}}/{{POLICY_NAME}}/{{EXECUTION_ID}}/{{RECORD_TYPE}}

This structure creates a separate folder for each datasource.

Use this option when you want storage organized based on data origin.

Organize results by execution time

To organize results by date or time, use the {{TIME:format}} variable.

You can define the format using Joda time patterns.

Example Template: {{POLICY_NAME}}/{{TIME:yyyy}}/{{TIME:MM}}/{{TIME:dd}}/{{EXECUTION_ID}}/{{RECORD_TYPE}}

This structure creates a year/month/day folder hierarchy.

Use this option when you need chronological storage for reporting, retention, or archival purposes.

Valid time formats

Format	Example Output	Valid
`yyyy-MM-dd`	2026-02-04	Yes
`yyyyMMdd`	20260204	Yes
`yyyy/MM/dd`	2026/02/04	Yes (creates path segments)
`yyyy-MM`	2026-02	Yes
`yyyy-MM-dd'T'HH-mm-ss`	2026-02-04T14-30-45	Yes

Format	Example Output	Reason
`HH:mm:ss`	14:30:45	Colon (:) is not allowed
`Z`	+0530	Plus sign (+) is not allowed
`yyyy-MM-dd'T'HH:mm:ss`	2026-02-04T14:30:45	Colon (:) is not allowed

Import and export of policies

When exporting policies:

Type	Behavior on Import or Export
System Default	Uses destination tenant default
Saved Configuration	Reused if exists, otherwise created
Custom	Preserved as-is

Limitations

Policies can override the storage base path — including directing results to a different bucket — by assigning a Persistence Configuration with a fully qualified URI as the Base Path. All configured buckets must be accessible via the same access credentials on the data plane and must reside in the same region as the data plane's global storage configuration. Per-bucket credential isolation is not supported.
Single credential set — all storage buckets used by a data plane must be accessible with the same access credentials. Per-bucket or per-domain credential isolation is not supported.
Same region required — all buckets must reside in the same region or endpoint as the data plane's global storage configuration. Cross-region writes are not supported.
ADLS bucket-level overrides — per-policy path overrides for Azure Data Lake Storage append to the global base path only. Container-level overrides for ADLS are not included in this release.
Spark History Server paths — the Spark History Server storage path is not configurable per policy in this release.