Take a Look at ADOC through the Latest Version!

Compute

• Simplified the Databricks registration flow by removing redundancies and streamlining the input sequence for a more intuitive onboarding experience.
• Enhanced the Databricks Cluster page with drill-down capabilities from cluster listings to job-level views, improved cost visibility, and smarter navigation for easier analysis.
• Introduced a unified Snowflake cost view, allowing users to select their preferred cost attribution type (Acceldata or Snowflake) and view it consistently across the platform.
• Added a job cost breakdown chart to the Databricks Job Details page, visualizing vendor cost components (VM, bandwidth, etc.) to support compute cost optimization.

Data Reliability

• Introduced integration support with Atlan in ADOC.
• Enabled WHERE clause support with {{{lower_bound}}} and {{{upper_bound}}} placeholders in SQL queries for Data Quality policies, allowing selective or incremental executions based on offset or timestamp values.
• Added an option for users to disable auto-classification tags.
• Refined the Tags page UI to enhance usability and clarity.
• Added support for non-English language data in Reliability policies.
• Centralized Spark Reader configuration and query construction in the control plane, reducing dependence on the dataplane.

Compute

• Enabled fetching of Databricks cost data from system tables for Azure Databricks data sources.

UI/UX

• Enhanced the Policy Summary Dashboard template with filtering options for Tags and Assets.

Data Reliability

• Enhanced crawling support in ADOC to fetch MongoDB views along with collections.
• Added support to create SQL Views on MongoDB views and collections.

Data Reliability

• Enhanced error records format for Snowflake Integration to standardized structure, simplifying Snowflake table creation and loading.
• The selection criteria for Selective processing and profiling strategies have been standardized for consistency. Both datetime-based and ID-based strategies follow the >= start and <= end format.
• The Pipelines Listing API allows filtering by asset ID, enabling the Asset Overview page in the ADOC UI to display related pipelines in a dedicated tab.
• The Asset Details page has been redesigned from a tab-based layout to a left-side navigation menu.

Pipelines

• Enhanced the Pipelines page UI for improved usability with new display fields, an upgraded list view, and an optimized details view featuring lazy loading and visual pipeline flow.

UI/UX

• ADOC enhanced accessibility compliance for the Login, Alerts Listing, and Alert Details pages to meet WCAG Level AA standards.

Common Services

• Updated API for RBAM, Domains, Resource Groups, and Admin Central.
• Enhanced RBAM to assign policy permissions to domain roles, refining access control and requiring users to review settings post-migration.
• Added AWS Service Principal support for Databricks.

Data Reliability

• Enhanced lineage UI by improving search, column highlighting, and updated table colors for better traceability.
• Automated Tableau lineage for seamless tracking of data flow from external sources to Tableau dashboards.
• Anomaly Detection in Data Quality allowing users to set anomaly sensitivity to low, medium, or high per rule while defining Data Quality policy rules.
• Added more control over Data Quality alerts by providing an option to disable warning notifications for a more focused alerting experience.
• Set annotations for crawler pods using KUBERNETES_POD_ANNOTATIONS.
• Switched from Klaxon to Jackson for improved efficiency and scalability.
• Upgraded Ktor to enhance security, boost performance, and improve backend stability and overall system efficiency.

Compute

• Enhanced data ingestion monitoring to track Snowflake ingestion workflows with improved dashboards.
• Optimized Query Studio Performance with refined columns and new Snowflake Cost field to enhance visibility into query costs.
• Enhanced SQL Warehouses insights for Databricks with detailed cost and performance analytics for Databricks SQL Warehouses.
• Excluded Iceberg table storage for more accurate Snowflake cost reporting.

UI/UX

• Added a filter to filter widgets by tags and type in the Widget Library.
• Enhanced widget metadata and descriptions for improved discoverability.
• Add pre-configured dashboards for users to get started quickly with ready-made templates and thumbnails.
• Expanded policy execution widgets to gain deeper insights with new execution summary widgets.
• White-Labeled Reliability reports for users to customize email reports by replacing the Acceldata logo.

Data Reliability

• Added separate permissions for policy execution in RBAC.
• Enhanced Row Check Measurement:
• Absolute Comparison: Users must specify both lower and upper bounds for validation.
• Relative Comparison: Bound inputs are no longer required; checks use the total row count.
• Incremental Strategy Support: Now compatible with incremental strategies for assets and custom SQL queries.

Data Reliability

• Updated profile settings to allow users to schedule asset profiling on a monthly basis.
• Improved webhook configurations to enable users to specify only the header key for sensitive data, with the actual values automatically fetched from the dataplane secret provider.
• Added capability to allow users to hover over an asset name on the Jobs page to see its complete hierarchy, simplifying the identification of the source asset.
• Extended OAuth support to Snowflake data sources when utilizing the Pushdown Engine, in addition to the previously supported Spark Engine.

Compute

• Added capability to allows users to configure a daily start time for database refreshes on the setup/edit page, ensuring consistent refresh schedules across environments.
• Introduced the ability to download data as a CSV for the specific page a user is viewing, with all applied filters accurately reflected.
• Added stock monitor to configure alerts to list tables created within a specified time frame by selecting a database and defining the duration (e.g., last 30 days).

Pipeline

• Optimized the pipelines APIs and UI to address latency issues during various pipeline flows.

UI/UX

• Added the capability to view and modify notification channels directly on the Data Reliability Reports page.

Common Services

• Introduced RBAM to enable enterprises to regulate access to assets like tables, reports, and policies, ensuring security, governance, and compliance.

Data Reliability

• Improved crawler reliability and reduced crawling time for large datasets, ensuring faster completion updates and seamless re-runs.
• Updated Manage Policies page to include sortable insights like quality score, execution status, mode, records processed, and duration of the latest run, along with total executions, 30-day average score, and open alerts count for richer insights.

Compute

• The Compute module’s navigation and data presentation have been streamlined.

Pipeline

• Enhanced the pipeline interfaces to load faster and more efficiently, by splitting and optimizing the /details API calls.

Data Reliability

• Added Pushdown data engine support for Databricks data sources, allowing data quality checks, profiling, and SQL operations.
• Integrated hourly scheduled jobs for Kafka data sources, capturing key metrics such as the number of messages, last message update time, and hourly message rates.
• Enhanced editing User Defined Template (UDT) by providing users with a comprehensive view of all locations where the UDT is currently in use.

Compute

• Added a new CSV export option for filtered recommendations,
• Enhanced ADOC system to retrieve service-specific rates directly from Snowflake metadata, ensuring more precise calculations. The system uses the user-provided CPC as a fallback if service-specific rates are not available.
• Additional widgets have been introduced to the Select Widgets section, allowing users to further customize dashboards.

User Experience

• Introduced filters—such as tag, policy type, and last result status—to Data Reliability widgets when creating a new dashboard.

Data Reliability

• Added support to view data lineage and relationships between Power BI and the detected external data sources used for its visualizations and reports.
• Implemented several UX improvements to ADOC including NLI feature flag visibility for specific policies, enhanced navigation, updated styling, UDF listing fixes, and alignment adjustments on the asset page.
• Added ability to export specific policy versions via API to a policy archive for import into higher enviroments.

Compute

• Added support for customizable dashboards allowing users to create personalized dashboards based on their requirements.
• Enhanced filters, including the date range, to remain intact in Compute when navigating back, drilling down, or refreshing the page, ensuring that users can seamlessly continue viewing or filtering the desired warehouse without the need to reapply their filter settings.

Pipeline

• Improved ADF pipeline user interface by enhancing parent-child pipeline visibility, adding a refresh button to reload only the live pipeline, and deprecating the pipeline-react flow feature.

Common Services

• Introduced an email OTP-based method for user invitations to improve security and address email rendering issues in certain email clients.

Data Reliability

• Added OAuth authentication support in ADOC for Alation cloud connections.
• Improved the Policy page for enhanced navigation, clarity, and functionality.
• Added support for Oracle data source users to enable and customize Data Freshness to track data cadence.
• Extended job queuing method for job execution with connection limits to support Hive (and MapR Hive, if different), HDFS, Kafka, MongoDB, and SQL Server.

Compute

• Enhanced the Snowflake Recommendations view to only show essential data by reducing data noise and keeping key columns. The view now lets users act on recommendations, simplifying optimization.
• Added Snowflake Organization Unit (OU) and Cost Center (CC) filters allowing users to use tag-based filtering on all pages, including Admin, Warehouse, and Performance. This allows more platform-wide data filtering and analysis.

Pipeline

• Integrated ADOC with Auto-Sys. Users can monitor AutoSys pipelines via ADOC.

Data Reliability

• Added dynamic column mapping for SQL rules and UDF rule sets allowing users to define them without pre-selecting columns.
• Added support for row count reconciliation using the Pushdown engine, improving efficiency by utilizing native data source capabilities.
• Introduced T-shirt sizing at the dataplane level, allowing users to manage Spark job resources for profiling and data quality, boosting performance.
• Added a queuing system to manage job execution (Data Quality, Reconciliation, Profiling) within connection limits, ensuring smooth sequential processing when limits are reached.

Compute

• Added a stock monitor to list tables created within a specified timeframe, allowing users to select databases and monitor newly created tables, improving data management.
• Compute Persona View has been set as the default page for the Compute section. Additionally, this update included UI improvements to NLI (Natural Language Interface).

User Experience

• Dataplane V1 has been deprecated and is controlled by a feature flag. New instances cannot be created, and the UI defaults to Dataplane V2, though existing V1 instances can still be deleted.

Data Reliability

• Added feature to allow users to create SQL Views directly on files in SFTP data sources, enabling aggregate checks and complex queries, streamlining processes previously reliant on visual views.
• Added option to allow users to manually add or override rules in Rule Set-driven policies and adjust rule weights without losing custom settings when the Rule Set is reapplied.
• Users can automatically propagate descriptions and metadata from bulk rules to policies and rules generated by a Rule Set.
• Added an update that installs the latest SAP HANA drivers, improving compatibility and performance while resolving issues caused by outdated drivers for seamless database operation.
• Improved integration enabling more efficient query execution in Synapse and Athena environments.

Compute

• Job Studio and All Purpose Clusters are generally available. Users are allowed to drill down to Job details from the Job Studio for improved monitoring.

User Experience

• A back button was added to the breadcrumb navigation for improved efficiency. Child elements now appear in a dropdown when there are more than two.
• Users now receive ADOC software incidents and maintenance updates. System downtime and maintenance are shown in real time in-product.

Data Reliability

• Introduced crawler runtime status to track the crawler’s live progress, from start to completion, on each card.
• Added Pushdown support for Big Query and SAP HANA enabling more efficient data processing.
• Enhanced the cadence feature to track hourly executions and failures, with improved schedules and architecture. It supports SAP HANA, MongoDB, and MSSQL.
• Improved performance for date-partitioned files in AWS S3, GCS, and ADLS reduce processing time with optimized path generation.

AI/ML

• Introduced an unified Text-to-Rules and Text-to-SQL modal that generates both rules and SQL queries from natural language inputs.

User Experience

• Revised visuals, color schemes, fonts, and layout to enhance readability and visual appeal across key ADOC sections.

Data Reliability

• Integration with Alation enables seamless data discovery and governance, allowing you to leverage Alation's cataloging and metadata management capabilities directly within the platform for improved data accessibility and collaboration.
• IRSA-based authentication is now supported on the ADOC platform, enabling secure access to AWS services for Data Reliability tasks without the need for static credentials, enhancing security and operational efficiency.
• The Catalog page now features a detailed Quality Score for each asset. Click on the percentage for a breakdown of how the score is calculated.

Compute

• The Node Size recommendation enhancement optimizes node recommendations for better accuracy and efficiency, allowing compute resources to automatically adjust to workload demands. Improved recommendations lead to enhanced performance and cost-efficiency in your Databricks environment, making autoscaling more intelligent and responsive.

User Experience

• Capability for users to create personalized dashboards by selecting and pinning widgets, setting custom views for different purposes like Cost Summary or Data Quality. The new framework also simplifies widget management and supports pre-built and dynamically assembled dashboards based on user preferences.

Data Reliability

• The NextGen data plane's introduction offers users enhanced performance and scalability to manage extensive data operations.
• Expansion of the incremental file processing capability for GCS and AWS S3 to include cross-account projects and enhance validation capabilities.
• Support for Azure Service Principal authentication was introduced for seamless integration with the Azure ecosystem and improved security.
• The integration with Azure Key Vault enhances data security by securely managing keys and secrets.
• ADOC Co-Pilot can now automatically generate policies, simplifying data governance. {Beta Feature}.
• Enhanced schema drift policies with additional configurations and improved detection algorithms.
• Automated metadata production for profiled tables and columns to enhance the ADOC-UI experience.
• Improved Data Reliability navigation and backend support.
• Enhanced file reader performance by leveraging native filesystem APIs and improving file cadence logic, resulting in lower latency in data intake workflows, especially for complicated folder structures and big data sets.

AI & ML

• Access the AI-driven rule recommendations to enhance data governance. {Beta Feature}

Data Reliability

• Introduced the Asset Similarity feature to quantify the resemblance between tables by analyzing column similarities across multiple tables, culminating in a holistic Table Similarity metric.| Beta Feature|
• Initiated OAuth implementation for enhanced security in ADOC Reliability, addressing compliance requirements for Snowflake integration.
• For selective execution, user input fields will automatically be populated based on the parameters of the last successful selective execution.
• The data reliability reports data no longer timeouts or loads slowly based on the given time range. The load process migration from PostgreSQL to Clickhouse improved this.

AI & ML

• Generative AI assistance improved with AI assistant for data and SQL rules.

Compute

• Interactive dashboards with conversational AI for Snowflake cost management, role-based user interaction, and targeted insights to address rising cloud prices and performance challenges.

Platform

• More data sources are supported by ADOC, improving its versatility and data observability. Amazon RDS PostgreSQL, Aurora PostgreSQL, Azure CosmosDB PostgreSQL, Aurora MySQL, RDS MySQL, and RDS MariaDB are now available.

Data Reliability

• Ensures continuous monitoring and alerting on critical assets. Users can customize notification preferences.

AI & ML

• Simplifies policy creation process by generating rules from natural language input.
• Provides an innovative solution for SQL validation and correction.

Compute

• Enhances issue tracking and management capabilities by synchronizing ADOC alerts with JIRA ticket status and priority changes.

Pipeline

• Introduced Automated Data Reliability in ADOC enabling you to create automations to execute policies and profiles during the execution of a pipeline.

Data Reliability

• Added support for cross-account access in cloud based file systems, allowing a principal in one account to access resources in a second account. This feature enhances flexibility and access control within the platform, implemented for GCP and AWS.
• Introduced an Alert Feedback Loop feature where users can tag alerts and provide feedback on alerts' usefulness and relevance. This feedback loop helps improve anomaly detection and model tuning for noise reduction.

Platform

• Enhanced user management with Single Sign-On (SSO) Group Integration using protocols like SAML, OIDC, and LDAP. This feature streamlines user access and provisioning, improving security and convenience within the platform.
• Improved auditing with new filters for Action Status and Method to show event outcomes and triggers. ADOC also logs client IP on the Audit details page for better security and compliance.

Data Reliability

• The Tag improvements in ADOC include enhanced user interaction with Tag Type Ahead, integration of Re-usable Auto-classification Tags for easier manual tagging, introduction of Missing Rule: Tag Rule for comprehensive tag verification, improved visibility of Column Tags, implementation of Tag Hierarchy or Groups for efficient management, and Complex Classification Rules enabling more precise tagging based on regex name and value matches.
• New data connectors are now available for Azure Blob Storage, ClickHouse, Power BI, MariaDB, MemSQL (Single Store) DB, and Tableau.

Platform

• Improved analysis features enable rapid identification of affected areas from alerts, with clearly defined dependencies and lineage provided within the application's dynamic Details page. This enhanced alert functionality includes detailed, niche-specific information to facilitate quicker issue resolution.
• Streamlined issue tracking and management through integration with JIRA, featuring API Token and OAuth authentication, and unidirectional ticket creation.
• Enhanced functionalities for audit logs including asynchronous downloads, SQL enhancements, bulk data insertions, and optimized filter fetching.

Data Reliability

• Improved security is achieved through the implementation of Databricks Service Principal authentication for both Azure and AWS environments, offering enhanced support.
• Users can now import and export Profile Anomaly or Data Cadence Policies, featuring advanced options for managing existing policies.

Compute

• Improved UI for consistent context in detail views.

Pipeline

• Upgraded SDK for better time monitoring and integration with external systems.

Infrastructure

• Enhanced visibility and observability in the data plane, including Spark history server integration.

Data Reliability

• Expanded anomaly detection capabilities to include semi-structured data types, enhancing data quality and integrity insights.
• Integration of MapR with SASL and non-SASL Hive settings, increasing ADOC's data source compatibility and versatility.
• Addition of AVRO file format support for data crawling, data quality, and profiling, facilitating seamless integration of AVRO-based assets.
• Revamped alert generation in Pipeline Services for system failures with customizable severity levels and notification options.
• Integration with Azure Active Directory for more secure authentication to Azure-related data sources.
• Integration of SAP HANA as a data source, leveraging its high-performance, real-time data processing capabilities for enhanced data reliability.
• New feature allowing users to configure alert generation for Data Reliability Policies, with options for alerts per execution or updating the same alert.

Compute

• New functionality for automating Databricks data transformation tasks, cluster management, and data quality checks.
• Improved filter usability with the addition of multiple search key components and context retention across navigations.
• Introduction of the Auto Renewal Token option in Databricks connectors for efficient token management and continuity.
• Performance improvements on the recommendation listing page with optimized sorting, filtering, and indexing.

Platform

• Enhanced feature access permissions in the RBAC screen for Dataplane, providing more refined control over user access.

Data Reliability

• Enhanced customization with the ability to rename notification channels for improved control and organization.
• New feature to rename custom or business assets directly in the SQL view, enhancing flexibility in managing business processes.
• Strengthened integration with Alation for improved data management and collaboration.
• Optimized data plane selection process for Spark jobs, focusing on proximity to the data source for better performance.
• Introduction of a widget in the Data Reliability module for more effective data visualization and monitoring.

Compute

• Resolved inaccuracies in the job and cluster status tab to display real-time information accurately.

Security and Integration

• Core libraries upgraded to enhance the overall security of the ADOC platform.
• Improved Webhooks Integration for automated ticket production from alerts, with enhanced URL input handling and JSON data transfer.

Permissions & Resource Management

• Big Query integration now works with read-only permissions, enhancing compliance with security norms and maintaining data integrity.
• New guidance for efficient resource allocation in Spark jobs, aimed at optimizing resource usage for Profiling or Data Quality tasks.

Data Plane Deployement

• Incorporation of the Spark History Server into ADOC data plane deployment, facilitating better job debugging experiences and simplifying the analysis and troubleshooting of Spark operations.

Data Reliability

• Integration of dynamic and static data filters in Bulk and Data Quality policies for precise policy creation. Now includes Spark SQL expressions.
• Simplifies transfer of rulesets, UDF templates, SQL expressions, and tags across environments. Enhances setup process and ensures uniformity.
• Administrators can now manage access to observed data based on roles or profiles, ensuring compliance with organizational data access policies.
• Users can now opt for alerts per execution or just for the first incident. Includes backend updates for dynamic reference ID generation and frontend alert generation UI toggles.
• Optimized incremental processing for Kafka, allowing incremental and selective data retrieval based on Kafka offsets. Users can define initial offset, enhancing data processing speed and efficiency.
• Multiple improvements in Profile Anomaly detection for a comprehensive anomalies overview, including user-friendly Profiling UI, Anomaly Policy Execution Flow, and updates to the Profile Setting Page. Enhanced data visualization and Profile Policy Execution UX. Also, configurable Time Series for prophet model, streamlined Reliability Report, and adjustments for Profile Anomaly Execution.
• Upgraded Cadence Chart to Confidence Chart for precise data display, introduction of Data Freshness chart for immediate insights, enhanced Metric formatting for consistency, refined Anomaly calculations for accuracy, Dynamic Data Cadence Display, and Granularity in Cadence Graph. Improvements in Data Visualization and bug fixes.

Platform

• Updated Admin Central for enhanced platform consistency. Includes optimized placeholder in the invite user flow, alignment of filter toolbar, standardized LDAP dropdown, and integrated date picker in Create API Key page.

Data Sources & Data Plane

• Azure Data Factory can be integrated with ADOC as a data source. Our platform includes a comprehensive dashboard that meticulously tracks various costs and resource utilization across factories, pipelines, and activities within Azure Data Factory.

Data Reliability

• Added the capability to activate a data freshness policy for an asset, which effectively identifies anomalies and breaches in SLAs.
• Added a spark SQL filter panel to selectively target specific data records that need to be evaluated for data quality issues.
• Introduced capability to use Google Cloud Secret Manager to store your secrets.
• Enhanced ADOC to offer the functionality of showcasing lineage details for assets originating from BigQuery data sources.
• ADOC provides you with an option to crawl either a single asset or select specific assets to add to an existing Snowflake, Redshift, or Databricks data source.

Compute

• Introduced capability in ADOC that empowers Databricks users to access real-time cost information for their Databricks resources on their Azure cloud platform.
• Added a dashboard that allows a provision to see user specific details such as the queries executed by the user and the corresponding costs incurred by them.
• Enhanced the Snowflake warehouse monitoring capabilities, enabling users to gain deeper insights into warehouse utilization, including productivity and wastage metrics.
• Enhanced the Compute Observability Setup page by providing an option to enable the creation of an external stage for storage integration.
• Added the option to use service principal identity to give access to users AWS or Azure Databricks resources.

Monitors & Alerts

• Introduced the following new monitors in Databricks Workspace: Running Clusters after Work Hours.
• Added the following ADF monitors that can be imported to the monitors list: Factory Cost, Pipeline Cost, and Failed Activities.