Release Notes 4.0.x

ADOC V4.0.1

Date: 10 February, 2025

Data Reliability

• Granular Permissions for Policy Execution: ADOC now supports separate permissions for policy execution as part of Resource-Based Access Management (RBAC). A new permission and role have been introduced, allowing administrators to enable or disable execution rights for users, providing greater control and security over policy enforcement. For more information, see User Management.
• Improved Row Check Measurement: To improve usability and flexibility, the Data Quality Policy has improved the Row Check measurement.
  • Absolute Comparison: Users must now specify both lower and upper bounds for validation.
  • Relative Comparison: When selecting a relative comparison, bound inputs are no longer required. Instead, the check is performed directly on the total row count of the table under evaluation.
  • Incremental Strategy Support: The updated Row Check measurement now works seamlessly with an incremental strategy, whether applied to a specific asset or a custom SQL query. These enhancements improve precision and simplify configuration for users defining data quality policies. For more details, refer to Row Checks.

• Fixed an issue where the cursor would move to the end of the SQL rule after each character typed; it now remains in the correct position.
• Implemented the ability to mark an asset as a Watched Asset, enabling users to track and filter assets more effectively on the Discover Assets page.
• Resolved an issue where users were unable to change the engine type from Spark to Pushdown or Pushdown to Spark in the new UI. This functionality is now enabled, allowing users to perform the action as needed.
• Resolved an issue where the Total Records Processed metric displayed as "0" for certain policies. It has now been updated to show N/A (Not Applicable), indicating that the metric does not apply in these cases and preventing confusion for users.
• Fixed an issue where the ADOC SQL View did not return valid results when querying an Oracle data source.
• Resolved an issue where the Add Lineage functionality was removed from the UI. It is now enabled, allowing you to define lineage as expected.
• Fixed an issue where the Groups page in Data Discovery navigation failed to load due to an error. The page now loads correctly, displaying the expected navigation options.
• Resolved issues where an anomaly flag was incorrectly triggered when the Actual value was within the Expected range, negative feedback could not be submitted, and the feedback message did not properly reflect a negative response when the thumbs-down was selected.
• Resolved a syntax error causing SQL View creation to fail for the MSSQL data source.
• Resolved an issue where DQ policies created using the custom SQL-based option were automatically archived after the next crawl if the query contained a primary key, requiring you to unarchive them after each crawl.
• Resolved multiple issues on the Pipeline page, ensuring filters are responsive, duplicate runs are no longer displayed, time zone specification is available for metrics, and only a single failure alert is triggered when detected.
• Resolved an issue where the partition-based incremental strategy with the sub-strategy day-month-year for profiling and DQ on the Hive data source was not working; also addressed UI issues by adding the day-month-year option in Profile/DQ Selective, updating DateFormat to DayFormat, and ensuring the UI displays dd/mm/yyyy markers as per the API data.
• Fixed an issue where column names were incorrectly renamed during selective profiling/DQ checks on Hive, causing errors related to non-existent columns, preventing successful completion of the checks.

ADOC V4.0.0

Date: 26 January, 2025

Data Reliability

• Monthly Asset Profiling Schedule: ADOC now allows users to schedule asset profiling on a monthly basis, enabling automated execution and ensuring timely evaluation of data. This feature reduces the need for manual intervention, streamlining the process and enhancing efficiency. For more information, see Profile Settings.
• Enhanced Security for Webhook Configurations: Users can now configure webhooks more securely by inputting only the header key for confidential information, while the actual values are automatically fetched from the dataplane secret provider. For more information, see Configuring the Webhook in ADOC.
• Asset Hierarchy Visibility on Jobs Page: Users can now hover over the asset name on the Jobs page to view its complete hierarchy, enabling easy identification of the source asset. For more information, see Jobs.
• Snowflake OAuth Support Update: ADOC now extends OAuth support to Snowflake data sources when utilizing the Pushdown Engine, in addition to the previously supported Spark Engine. For more information, see ADOC OAuth Implementation.

Pipeline

• Enhanced Pipelines UI and Backend Performance: Optimized the pipelines APIs and UI to address latency issues during various pipeline flows. Backend changes have been implemented, and UI updates include:
  • Improved alert details population for enhanced data visibility.
  • Fixes to the pipeline editing flow for seamless user experience.

Compute

• Database Refresh Scheduling: Added the ability to configure a daily start time for database refreshes on the setup/edit page, ensuring consistent refresh timing across environments. This enhancement allows you to set preferred refresh times, enabling updates to align with your workflow and avoid timing resets due to changes in data source configuration. For more information, see Snowflake Observability Set Up.
• Export Raw Data from Query Studio Page: Introduced the ability to download data as a CSV for the specific page you are viewing, with all applied filters accurately reflected. This ensures precise data exports tailored to your selected view and filter criteria. For more information, see Snowflake Query Studio and Databricks Query Studio.
• Custom Alerts for Table Creation: Added the ability to configure alerts that provide a list of tables created within a specified time frame. You can select a database and define the duration (e.g., tables created 30 days ago) to monitor and track database activity effectively. For more information, see Snowflake Stock Monitors.

UI/UX

• Enhanced Notification Channel Management: Users now have the ability to view and modify notification channels directly on the Data Reliability Dashboard, offering greater flexibility and control. This feature simplifies debugging when reports are not sent, allowing users to review schedules or settings and make adjustments as needed.
  • Edit Notification Channels: Modify existing notification channels without the need to create new ones.
  • Improved Troubleshooting: Quickly identify and resolve issues with report schedules or settings.

This update streamlines the process of managing notification channels, enhancing user efficiency and overall experience.

For more information, see Editing Notification Channels for a Report.

Common Services

Introducing Resource-Based Access Management (RBAM): RBAM empowers enterprise users to regulate access to various assets such as tables, reports, and policies meeting critical security, governance, and compliance requirements. Core Features:

• Resource Groups: Administrators can now group resources (e.g., assets, reports, policies, dashboards, and more) into resource groups, enabling streamlined and efficient access control.
• Domain for Access Management: RBAM introduces domains, allowing users to specify the resources that can be accessed and the users who can access them. This feature gives enterprises greater control over their data and resources while supporting distributed management.
• Enhanced Control: With granular permissions, RBAM ensures that users can only see and interact with the resource relevant to their roles, improving security and efficiency.

This feature paves the way for more secure, scalable, and efficient resource management, making valuable additions for enterprises operating in complex data environments.

For more information, see Resource Groups, User Management and Domain Management.

Data Reliability

• Resolved the issue where Rule Sets could create policies with invalid names, including names containing spaces.
• Fixed an issue where users were not prompted to input values for function variables (e.g., {{{input_value}}}) used in SQL filters during the creation of a reconciliation policy).
• Fixed an issue where changes made to alerts and notification channel configurations were not being saved when editing a policy.
• Resolved the issue where users were unable to perform selective operations on profiles or execute Data Reliability policies on Kafka topics.
• Resolved discrepancies in data quality scores and policy execution counts across pages.

Compute

• Fixed an issue where job clusters continued execution beyond the configured long-running cluster threshold. Clusters now terminate as expected once the threshold is exceeded.

UI/UX

• Resolved an issue where the Control Plane would hang or display a 404 error when a user without the correct role permissions attempted to log in. The system now properly displays a permissions error to inform the user of the access issue.

• Within the Freshness Trend chart found in the Data Cadence tab, the upper bound, lower bound, and change in row count metrics are exclusively accessible for time filters such as today, yesterday, and those based on an hourly time frame.
• Anomaly detection is not supported for nested columns of an asset.
• When the refresh token time expires for ServiceNow OAuth Integration, the status remains as configured instead of changing to expired.
• Dataproc is only supported on GCP as the cloud provider for data plane installation.
• Glossary window fails to load data due to Dataproc APIs failure.
• The Smart tag feature is currently available only for Kubernetes-driven Spark deployments.
• When a 2FA credential is activated, the credentials do not work.
• User-specific usage data for a specific day may not be accurately displayed or retrieved.
• Issue with GCP test connections.
• Profiling in BigQuery fails for tables when the partition filter is enabled.
• DQ Policy fails sometimes due to an inability to verify a space column with special characters.
• Unable to pick a reference asset on a Data Policy template when defining a Lookup Rule.
• The data lineage view for job runs is not visible on Databricks' Job Run Details page.
• If all values are null for primitive string and complex data types, profiling will occur, but the data type will be unknown.
• All failed events are not captured in the Audit Logs page.
• When performing Incremental Data Quality (DQ) checks with Datetime2 andDateOffset formats in Synapse, if no new records are added, the system processes the last record instead of skipping processing.

• Initial ADOC Setup
• Architecture