Release Notes 3.14.0

Date: 18 November, 2024

Data Reliability

• Power BI Data Lineage Support: With Power BI integration, ADOC now establishes data lineage by detecting external data sources used in Power BI reports. After both the external source and Power BI are registered and crawled, our platform detects relationships and finds lineage between them, enhancing cross-system data visibility. For more information, see Microsoft | Power BI.
• Implemented Several UX Improvements: ADOC now includes NLI feature flag visibility for specific policies, enhanced navigation, updated styling, UDF listing fixes, and alignment adjustments on the asset page.
• Ability to Export Specific Policy Versions via API: Users can now export specific policy versions to a policy archive for import into higher environments. This feature allows teams to select and promote particular policy versions without exporting the most recent version, increasing development flexibility. For more information, see API Docs.

Pipeline

• Improved ADF Pipeline UI:
  • Enhanced Parent-Child Pipeline Visibility : Users can now view parent pipeline information within child pipelines in the ADF Pipeline UI. This improvement allows for better navigation and understanding of pipeline hierarchies. For more information, see Pipeline Home.
  • Added Refresh button for Live Pipeline: The pipeline run details page now features a new refresh button for live pipelines. This enables users to monitor the progress of live pipelines in real time without having to reload the entire page. For more information, see Pipeline Graph.
  • Deprecated Obsolete Feature Flag: The Pipeline-React Flow feature flag, along with its associated legacy code, has been deprecated and removed from the user interface. The pipeline canvas now utilizes the latest components by default, streamlining the interface and improving performance.

Compute

• Customizable Dashboards: The Dashboard allows you to create, customize, and manage multiple dashboards based on your requirements. You may add and arrange widgets to display key insights and make a personalized dashboard your default view. Whether tracking Cost Summary or Data Quality, the easy structure ensures efficient widget administration, enabling you to quickly access the information that is most important. For more information, see Dashboard.
• AI-Powered Query Advisor: The Query Advisor, an AI-powered tool, evaluates and optimizes SQL queries. It detects inefficiencies such as high execution costs, long runtimes, large data scan volumes, and makes actionable recommendations to improve query performance. By addressing these challenges, the Query Advisor promotes optimal resource use and overall system efficiency, especially in Snowflake contexts. For more information, see Query Advisor.
• Persistent Filter Settings: Filters, including the date range, now remain intact in Compute when navigating back, drilling down, or refreshing the page. This enhancement ensures users can seamlessly continue viewing or filtering the desired warehouse without the need to reapply their filter settings.

Common Services

• Email Invitation Enhancement with OTP Mechanism: Introduced an email OTP-based method for user invitations to improve security and address email rendering issues in certain email clients. For more information, see Initial ADOC Setup.

Data Reliability

• Resolved an issue where downloading the configuration JSON for the schema drift policy resulted in a 500 Internal Server Error. This download function now operates successfully.
• Resolved an issue where downloading the configuration JSON for the reconciliation policy resulted in a 400 Error with the message Cannot process more than one rule. This function now works correctly from both the policy listing and view policy pages.
• Fixed an issue where executing the row count reconciliation policy with a dynamic SQL Spark filter resulted in an error. This policy now executes successfully with dynamic SQL filters.
• Resolved an issue where exporting the JSON config for an earlier policy version incorrectly downloaded the latest version instead of the selected version.
• Fixed a permission issue where data cadence metrics were not being collected for SQL Server, preventing the evaluation of Data Freshness policies. These metrics are now correctly collected and policies are evaluated as expected.
• Resolved an issue where pipelines failed to appear if duplicate entries existed in the job metadata JSON. Pipelines now display correctly regardless of duplicate entries.
• Resolved an issue where two 'Schedule Profile' tabs appeared for Hive data sources, causing confusion in setting profile schedules. Only a single tab now displays for scheduling profiles as intended.
• Resolved an issue where the Snowflake Crawler failed with No active warehouse selected due to not using the specified Snowflake Role warehouse in Acceldata connection settings. The crawler now correctly applies the specified warehouse, ensuring successful execution.

Pipeline

• Fixed several user interface issues, such as correcting breadcrumb navigation, ensuring proper icon displays on hover for clickable buttons, and provided support for multi-level breadcrumbs. These changes enhance the overall user experience when interacting with pipeline details.

Compute

• Resolved compatibility and performance issues by replacing the Elasticsearch client with the OpenSearch client.

• Within the Freshness Trend chart found in the Data Cadence tab, the upper bound, lower bound, and change in row count metrics are exclusively accessible for time filters such as today, yesterday, and those based on an hourly time frame.
• Anomaly detection is not supported for nested columns of an asset.
• When the refresh token time expires for ServiceNow OAuth Integration, the status remains as Configured instead of changing to Expired.
• Dataproc is only supported on GCP as the cloud provider for data plane installation.
• Glossary window fails to load data due to Dataproc APIs failure.
• The Smart tag feature is currently available only for Kubernetes-driven Spark deployments.
• When a 2FA credential is activated, the credentials do not work.
• User-specific usage data for a specific day may not be accurately displayed or retrieved.
• Issue with GCP test connections.
• Profiling in BigQuery fails for tables when the Partition filter is enabled.
• DQ Policy fails sometimes due to an inability to verify a space column with special characters.
• Unable to pick a reference asset on a Data Policy template when defining a Lookup Rule.
• The data lineage view for job runs is not visible on Databricks' Job Run Details page.
• If all values are null for primitive string and complex data types, profiling will occur but data type will be Unknown.
• All failed events are not captured in the Audit Logs page.

• Initial ADOC Setup
• Architecture