Release Notes 4.2.x

ADOC V4.2.2

Date: 9 April, 2025

• Fixed the issue where query lineage was not displayed for assets when using Snowflake OAuth authentication. The query lineage now correctly populates upstream and downstream relationships, ensuring visibility into data lineage.
• Removed redundant lines related to stdout/stderr logging from the Databricks global init script. These lines were previously causing confusion during manual customer environment setup, even though logging was already disabled. This change ensures clarity and confirms that no logs flow from customer systems to ADOC.
• Fixed the issue causing the Discover Assets page to display incorrect counts for Data Quality (DQ) policies attached to assets. The policy count now accurately reflects all associated DQ policies. Note For previously created policies, the accurate count will update after policy execution.

ADOC V4.2.1

Date: 7 April, 2025

Data Reliability

• MongoDB Crawling now Supports Views: The ADOC platform now crawls both collections and views from MongoDB data sources. Previously, only collections were detected. This update ensures metadata for both objects is fetched and listed separately. For more information, see MongoDB.
• Support for Creating SQL Views on MongoDB Collections and Views: You can now create SQL views on MongoDB collections and views using MongoDB aggregation pipeline queries. This enables you to define virtual assets, rename fields, and apply complex filters—such as extracting specific elements from arrays—without modifying the underlying MongoDB data. For more information, see Creating SQL View for MongoDB Data Source.

• Fixed the issue where the search bar in the DR Reports page was not filtering results based on the entered asset name. Now, when users search for an asset, the report correctly updates to display data related to the searched asset.
• Fixed the issue where selecting Azure Data Factory (ADF) as the pipeline source did not correctly filter the datasource list or pipeline runs. Now, only ADF-related datasources are displayed, and newly configured ADF datasources appear immediately after selection. Additionally, when a specific Data Factory is selected, only pipeline runs related to that Data Factory are displayed. The filter now functions independently for each Data Factory, even when multiple exist within the same datasource. The API ensures that only ADF-specific datasources and relevant pipeline runs are returned.
• Fixed the issue where pipeline lineage in Acceldata was displayed incorrectly when an IF clause or specific conditions were used in Azure Data Factory (ADF) pipelines. Now, the lineage is accurately generated and displayed.
• Fixed the issue where the Data Freshness policy displayed as Not Yet Executed for Snowflake assets, even when enabled. Now, the policy correctly shows execution details and the Data Freshness score after configuration.
• Fields such as Rows Scanned, Rows Failed, and Warning Rows have been removed from Data Drift Policy results, as they were not applicable and always appeared as null.
• Fixed the issue where sub-level assets (schemas, databases, tables) in the Navigator panel on the Discover Assets page were sorted in descending alphanumeric order. Now, they are correctly sorted in ascending alphanumeric order.
• Fixed the issue where the Asset page was slow to load and did not display Asset scores. Now, the page loads efficiently, and the scores appear as expected.
• Fixed the issue where clicking on Sub-Lineage caused the page to fail. Now, column-level details are displayed correctly for supported sources.
• Fixed the issue where label filtering did not work in the new Discovery UI and Reliability Explorer reports, while it functioned correctly in the old UI. Now, filtering on a label properly selects the correct assets across all dashboards.
• Fixed the issue where execution details were missing in alert notifications for Data Quality policies sent via email and Teams. Now, the alerts include the success rate of each rule, ensuring consistency with the details available on the Policy Summary page.
• Fixed the issue where query lineage required create permissions on production projects in BigQuery. Now, lineage can be defined using only create temp table access, ensuring that customers can maintain security constraints while still enabling query lineage visibility.
• Fixed the issue where Airflow-generated pipeline runs older than 45 days were not deleted during the upgrade to 4.2 pipeline code. Now, the upgrade process ensures that all pipeline runs older than 45 days are properly removed, consistent with the expected behavior.
• Fixed the issue where users were unable to create SQL views for a Redshift datastore due to the missing "Select a database" option. Now, the UI correctly provides an option to select a database, allowing users to create SQL views seamlessly.
• Fixed the issue where the lineage query was executed in the US region instead of the designated EU region, despite the Kubernetes cluster using an EU-based BigQuery instance. Now, the lineage query executes only within the same region as the dataset, ensuring compliance with data governance policies and GDPR requirements.

ADOC V4.2.0

Date: 27 March, 2025

Data Reliability

• View Pipeline for Selected Asset: The Pipelines Listing API now allows filtering by asset ID, enabling the Asset Overview page in the ADOC UI to display related pipelines in a dedicated tab. This enhancement leverages the existing pipeline listing page for a more contextual and streamlined experience. For more information, see Pipelines.
• Enhanced Asset Details UI for Improved Navigation: The Asset Details page has undergone a UI enhancement, transitioning from a tab-based structure to a left navigation menu. This improves usability by enabling users to quickly access different sections and perform actions more efficiently. For more information, see Asset Details.
• Standardized Persistence Path Format for Data Quality and Reconciliation Policies: The persistence path structure for storing policy outputs—such as error records, good records, and policy summaries—has been standardized. The new format organizes records by policy name and date partition. Existing structures for older data planes remain unchanged. For more information, see Create Data Quality Policy.
• Standardized Selective Strategy Execution for Data Quality Policies and Profiling: The selection criteria for selective processing and profiling strategies have been standardized for consistency. Both datetime-based and ID-based strategies now follow the >= start and <= end format.

Pipelines

• Enhanced Pipeline UI for Better Usability and Insights: The Pipeline UI now includes:
  • New Display Fields: Pipeline Source (e.g., Airflow, DBT, Spark)
  • Improved List View: Status column, clickable alerts, sortable columns, and a sparkline area chart for recent runs
  • Enhanced Pipeline Details: Optimized layout, lazy loading for better performance, and a visual representation of pipeline flow. For more information, see About Pipelines.

UI/UX

• Accessibility Compliance (WCAG Level AA): ADOC has significantly improved accessibility compliance for the Login, Alerts Listing, and Alert Details pages. Enhancements include full keyboard navigation, appropriate ARIA labels and roles for screen readers, improved color contrast, and consistent navigation order. These updates ensure a more accessible experience for users, aligning fully with WCAG Level AA standards. For more information, see Accessibility Compliance.

Common Services

• Updated API for RBAM, Domains, Resource Groups, and Admin Central: The API has been updated to include detailed information on RBAM (Role-Based Access Management), Domains, Resource Groups, and Admin Central. These updates enhance clarity, improve usability, and ensure that users have the latest guidance on managing access controls and resource configurations effectively. For more information, see API Authentication and Authorization .
• RBAM: Policy Permissions Migrated to Domain Roles: Policy permissions have been moved to domain roles for finer access control. System-defined roles now include policy permissions:
  • Resource Viewer: View policies
  • Resource Editor: Modify, view, and execute policies.
  • Resource Owner: Create, modify, view, and execute policies.

Custom roles with view asset permission now include view policy, while those with modified asset permission gain full policy access. Users must review and adjust permissions post-migration. For more information, see Authorizations in ADOC.

• AWS Service Principal Support for Databricks Onboarding: ADOC now supports using Service Principals for the compute-enabled AWS Databricks data source integrations. This enhancement now streamlines the integration workflows, offering improved security and simplified configuration flows for users leveraging Databricks and AWS environments. For more information see, Databricks.

Data Reliability

• Resolved the issue where Row Check rules in absolute mode required both lower and upper bounds to be entered in Data Quality (DQ) policies. The update restores the ability to specify only a lower bound without enforcing an upper limit, preventing users from having to input excessively large values.
• Fixed the problem where policies created before version 4.1.0 without notifications or alerts enabled were incorrectly having the ‘Notify on Warning’ toggle turned on after the upgrade. The update ensures that only policies with alerts enabled prior to 4.1.0 will have the toggle turned on, while those without notifications remain unaffected.
• Addressed the issue where the Rows Scanned, Rows Failed, and Warning Rows fields were displaying as null in the Data Drift Policy. The update ensures that these fields are not shown when they are not applicable, thereby preventing any ambiguity.
• Fixed the issue where data types were inconsistently changing when profiled using Spark and Pushdown. The update ensures that data types remain consistent across both profiling methods, maintaining accuracy and reliability in data profiling results.
• Fixed the problem where clicking on an asset not linked to a catalog item in the new lineage view caused the screen to go blank. Now, unknown assets are displayed in grey, and clicking on them either has no effect or shows a card with available metadata, This ensures the lineage view remains visible without requiring a page refresh.
• Addressed the issue where using an SQL Template to import an SQL Rule prompted users multiple times for the same input token. Now, when a validation UDF contains a repeated input token, users are prompted only once per unique token, ensuring a consistent experience similar to UDF validation and UDF rule usage.
• Fixed the issue where User-Defined Functions (UDFs) failed when passing a value instead of a column for evaluation. The update ensures that expression-based UDFs with input tokens correctly process fixed input values, allowing policies to execute successfully without misinterpreting values as columns.
• Resolved the Azure Databricks crawler job failing when configuring Service Principal authentication with both Cost Op and DR are enabled. Previously, the Test Connection validated only the API endpoint but did not check the JDBC URL, leading to a successful test despite an invalid JDBC connection. The update ensures that the Test Connection now validates both API and JDBC URLs and provides an error if either fails, preventing crawler job failures due to incorrect configurations.
• Fixed the issue where users with the required permissions were unable to create data sources in ADOC. The update ensures that permission validation correctly applies to the user configuring the data source, allowing all authorized users to create data sources without dependency on the dataplane creator’s permissions.
• Resolved the issue where Redshift transaction logs and lineage data were missing after creating a Redshift data source. The update ensures that the Query Analyzer service correctly extracts lineage when using Username/Password authentication.
• Fixed the issue where ADF pipeline runs were not visible when using a managed identity with Data Factory read permissions. The update ensures that pipeline runs are correctly displayed when an ADF data source is added using a managed identity, provided it has the necessary read permissions.
• Addressed the issue where the Total Policies widget UI/Reporting page displayed an incorrect DQ policy count. The update ensures that all DQ policies, including those on SQL views, are accurately counted and reflected in the Total Policies widget, eliminating discrepancies between the selection and the actual number of policies.
• Fixed the issue where the Data Processed field in the policy summary page was undefined. The update ensures that the correct amount of data processed is accurately retrieved and displayed.
• Resolved the problem where the 'Disable Toggle' for Policy Management was missing in the new UI. The update ensures that policies can be enabled or disabled directly within the new UI.
• Fixed the issue where SQL-based DQ policies in pushdown mode did not allow column selection after SQL validation. The update ensures that, once SQL is validated, users can select available columns and configure rules as expected, resolving the issue where no columns were displayed for selection.
• Fixed the issue where incremental profiling and incremental DQ policies failed for ADLS date-partitioned Parquet files. The update ensures that the incremental strategy correctly picks files based on the configured pattern, lookback days, and time zone settings, allowing profiling to process data as expected.
• Fixed the issue where incremental profiling and policies failed for ADLS date-partitioned Parquet files after the first execution. The update ensures that subsequent incremental profiling runs function correctly, whether triggered manually or scheduled, allowing daily refresh cycles to process data as expected.
• Fixed the issue where error messages were not handled gracefully when a file was missing in ADLS. The update ensures that, if a file is not found, the system provides a clear and informative message instead of an unhandled error.

Compute

• Fixed the issue where selecting a query in Snowflake Query Studio displayed incorrect query details. The page title and URL reflected the selected query ID, but the summary section showed details from a different query. This issue has been resolved, ensuring that the drill-down to query details now correctly displays the text and summary of the selected query.
• Fixed the issue where the Databricks Datasource widget did not properly display the connection status on the data source listing page.
• Fixed an issue where Databricks Compute's cost data was not being populated. The update ensures that Databricks and vendor costs are correctly retrieved when utilizing the Azure service principle. Additionally, job details in Job Studio now show success, failure, and active status, and the Databricks Compute recommendation list is no longer empty.
• Resolved the issue where the new UI could not save changes to a policy after switching the data processing engine (Spark/Pushdown). Additionally, addressed the issue where the Performance Trend widget was unable to display results based on data availability, even though it had at least seven days of history available.
• Fixed the issue where users were unable to remove or update selected filters in the Data Source filter on Custom Dashboards. Users can now successfully add, remove, or modify filter selections as expected.
• Fixed the issue where a cost mismatch was observed between Databricks (DBX) costs and vendor costs due to UTC time zone discrepancies. The update ensures that cost calculations now align correctly with ADOC, eliminating variances and restoring customer confidence in the reported data.

• Within the Freshness Trend chart found in the Data Cadence tab, the upper bound, lower bound, and change in row count metrics are exclusively accessible for time filters such as today, yesterday, and those based on an hourly time frame.
• Anomaly detection is not supported for nested columns of an asset.
• When the refresh token time expires for ServiceNow OAuth Integration, the status remains as configured instead of changing to expired.
• Dataproc is only supported on GCP as the cloud provider for data plane installation.
• Glossary window fails to load data due to Dataproc APIs failure.
• The Smart tag feature is currently available only for Kubernetes-driven Spark deployments.
• When a 2FA credential is activated, the credentials do not work.
• User-specific usage data for a specific day may not be accurately displayed or retrieved.
• Issue with GCP test connections.
• Profiling in BigQuery fails for tables when the partition filter is enabled.
• DQ Policy fails sometimes due to an inability to verify a space column with special characters.
• Unable to pick a reference asset on a Data Policy template when defining a Lookup Rule.
• The data lineage view for job runs is not visible on Databricks' Job Run Details page.
• If all values are null for primitive string and complex data types, profiling will occur, but the data type will be unknown.
• All failed events are not captured in the Audit Logs page.
• When performing Incremental Data Quality (DQ) checks with Datetime2 andDateOffset formats in Synapse, if no new records are added, the system processes the last record instead of skipping processing.

• Initial ADOC Setup
• Architecture