Release Notes 3.13.0

Date: 28 October, 2024

Data Reliability

• Implemented Data Freshness Policy for Oracle Data Sources: Users can now enable and configure the Data Freshness setting for Oracle data sources, allowing them to track data cadence. By default, the Data Freshness policy runs automatically every hour. DBA (Database Administrator) privileges are required for successful policy execution. For more information, see Data Freshness Policy.
• Added Support for OAuth Authentication for Alation: ADOC now supports connecting to the Alation cloud platform using OAuth authentication. Users can enable OAuth and provide the required Client ID, Client Secret, and Access Token Duration, as configured by the Alation Server Admin. To authenticate via OAuth in ADOC, the Alation Server Admin must assign a minimum role of Catalog Admin while setting up the authentication. For more information, see Alation.
• Data Plane Support for Incident Creation: Incident creation requests can now be routed through the data plane when network restrictions, such as proxies or firewalls, prevent direct access to your application. This allows you to comply with network policies while maintaining seamless incident management. For more information, see Data plane Support for Incident Creation Using Webhook.
• Extended Queueing Mechanism for Job Execution with Connection Limits: The queuing mechanism for job execution with connection limits has now been extended to support additional data sources, including Hive (and MapR Hive, if distinct), HDFS, Kafka, MongoDB, and SQL Server. This feature, originally introduced for managing manual and scheduled job executions—such as Data Quality, Reconciliation, and Profiling—ensures that concurrent connection limits at the data source level are respected. When the maximum number of connections is reached, jobs are queued and executed in sequence, preventing connection limit or timeout failures. For more information, see Concurrent Connections and Queueing Mechanism

Pipeline

• Autosys Integration Support: ADOC is now integrated with AutoSys. It allows users to monitor and observe AutoSys pipelines directly from the ADOC platform. This feature improves data observability by visualizing the job dependencies, monitoring job executions, and managing pipelines within a unified interface. For more information, see AutoSys.

Compute

• Enhanced Snowflake Recommendations View: The Snowflake Recommendations View has been refined to display only the most relevant information by removing the unnecessary data noise and retaining key columns. Users can now directly perform actions based on recommendations within the view, simplifying the optimization process. For more information, see Recommendations.
• Enhanced Tag-Based OU/CC Filtering for Snowflake: Tag-based filtering is now enabled within Organization Unit (OU) and Cost Center (CC) filters for Snowflake and extended to all pages - including Admin, Warehouse, and Performance. This provides greater flexibility in data filtering and broader data analysis across the platform. For more information, see Snowflake Admin, Snowflake Performance, and Snowflake Warehouse.

Data Reliability

• Resolved the issue in Snowflake pushdown where backslashes in regex patterns were incorrectly escaped, causing all rows to fail the match. Regex patterns are now correctly processed, allowing accurate row matching during pushdown execution.
• Resolved the issue where the crawler intermittently failed during large dataset crawls, specifically when pushing asset batches from the data-plane to the control plane. This has now been fixed.
• Fixed the issue where the pipeline run dropdown was not scrollable, limiting visibility to the last 10 runs. This has been resolved to allow full scrolling through all pipeline runs.
• Resolved an issue where the crawler did not retrieve all pipelines when crawling larger Azure Data Factory data sources. All pipelines are now successfully pulled during the crawl.

Compute

• Resolved an issue where the Recommendations page displayed only the count of Data Quality recommendations without listing the details. The page now correctly shows detailed recommendations.

• Within the Freshness Trend chart found in the Data Cadence tab, the upper bound, lower bound, and change in row count metrics are exclusively accessible for time filters such as today, yesterday, and those based on an hourly time frame.
• Anomaly detection is not supported for nested columns of an asset.
• When the refresh token time expires for ServiceNow OAuth Integration, the status remains as Configured instead of changing to Expired.
• Dataproc is only supported on GCP as the cloud provider for data plane installation.
• Glossary window fails to load data due to Dataproc APIs failure.
• The Smart tag feature is currently available only for Kubernetes-driven Spark deployments.
• When a 2FA credential is activated, the credentials do not work.
• User-specific usage data for a specific day may not be accurately displayed or retrieved.
• Issue with GCP test connections.
• Profiling in BigQuery fails for tables when the Partition filter is enabled.
• DQ Policy fails sometimes due to an inability to verify a space column with special characters.
• Unable to pick a reference asset on a Data Policy template when defining a Lookup Rule.
• The data lineage view for job runs is not visible on Databricks' Job Run Details page.
• If all values are null for primitive string and complex data types, profiling will occur but data type will be Unknown.
• All failed events are not captured in the Audit Logs page.

• Initial ADOC Setup
• Architecture