Architecture
The architecture is a high-level overview of the Data Plane, Control Plane, Installation, and Security. These are the critical components of ADOC infrastructure. Proper installation and security procedures are critical to ensuring that the ADOC operates efficiently and safely.
ADOC Architecture Overview
The Acceldata Data Observability Cloud (ADOC) is an integrated system that provides reliable data observability solutions. The design is purposefully divided into two distinct settings - the Control plane which is managed by the Acceldata and the Dataplane - to enable effective data handling and monitoring.
The present architectural diagram depicts the dynamic interplay between these two settings by highlighting the major components and their relationships.
The following diagram illustrates the architecture of ADOC:
ADOC Architecture
Control Plane
This environment represents the nerve center of the ADOC platform, powered by Amazon Web Services (AWS). It encapsulates a suite of AWS services meticulously orchestrated to host and run the ADOC platform, with Acceldata being accountable for the management of the cloud infrastructure.
Control Plane
Key Features of Control Plane
Managed by Acceldata, the control plane is the epicenter for administrative operations. It houses the core microservices, orchestrating tasks, managing configurations, and overseeing communication across the platform. It encompasses several critical microservices.
- Virtual Private Cloud (VPC): The backbone of this network architecture, a VPC provides logically isolated sections of the AWS cloud where resources can be launched in a defined virtual network.
VPC Layer - Control Plane
- Zones: The ADOC Control Plane's design is divided into zones that reflect distinct regions of the VPC. These zones aim to improve the system's resilience and availability. By compartmentalizing resources, each zone can run semi-independently, resulting in efficient load distribution and robust fault tolerance. Zone 1, Zone 2, Zone 3,each numbered zone represents a subnet or cluster within the VPC. This multi-zone architecture guarantees that services are distributed across multiple physical locations, lowering the chance of simultaneous outages and providing geographic redundancy. Workloads are balanced among these zones to improve speed and minimize latency for end users.
Zones - Control Plane
- Public Subnets: These subnets are part of the cloud network visible to the public internet, hosting resources that need to be accessed directly, such as the Cloud User Interface.
- Private Subnets : Contrary to public subnets, these are isolated sections of the cloud network that house resources requiring additional security, limiting public access.
Subnets Layer - Control Plane
- Load Balancers: These are strategically placed to distribute the incoming network traffic across multiple servers, preventing any one server from becoming a bottleneck, thus ensuring efficiency and high availability.
Load Balancers - Control Plane
- Nodes: Nodes are individual instances or servers in a subnet that make up a bigger network infrastructure. Each node acts as a separate entity capable of processing, storing, and forwarding data. The connections between nodes are network routes that enable communication and data exchange. These connections are critical for a network's dispersed activities, guaranteeing that each node can work together to accomplish collective tasks or services.
Nodes - Control Plane
- Secure Channel: A secure, SSL-encrypted channel bridges the Control Plane with the Data Plane ensuring that data transfers maintain integrity and confidentiality.
Secure Channel
- Datastore Search: The Search Datastore is a specialized database within the ADOC platform designed for efficiently handling and executing search queries across vast datasets. This datastore indexes data in a way that optimizes search operations, allowing for rapid querying and retrieval. It's engineered to support complex searches, including full-text and multi-parameter searches, facilitating immediate access to relevant data for analytics and monitoring tasks.
Datastore Search - Control Plane
- Cloud Storage: Cloud Storage within the ADOC platform represents scalable and secure data storage solutions in the cloud. It is used for persisting large volumes of data generated and used by the platform, including logs, metrics, and configuration information. Cloud Storage ensures data durability and high availability, making it accessible for processing and analysis by other components within the platform. It supports the platform's infrastructure by providing a reliable foundation for storing and managing data that is crucial for observability and operational intelligence.
Cloud Storage - Control Plane
- Message Bus: The Message Bus in the ADOC Control Plane architecture is a powerful communication system that facilitates the exchange of information between different components and services within the platform. It acts as a central conduit for message passing, ensuring that data flows seamlessly and efficiently across the system. The Message Bus is designed to handle high-throughput, low-latency messaging, which is essential for real-time data processing and event-driven architectures. It provides a reliable and scalable way to decouple system components, allowing for flexible and maintainable codebases.
Message Bus - Control Plane
- Database: The Database component serves as the persistent storage solution within the ADOC Control Plane architecture. It is responsible for securely storing all the platform's configuration data, metadata, and the observability data collected from various sources. The database is optimized for high performance and availability to ensure that data retrieval and storage operations can keep pace with the demands of the observability platform. It plays a vital role in data management, supporting transactional operations, complex queries, and providing a foundation for analytics and reporting functionalities.
Database - Control Plane
These components work together to create an ecosystem that not only ensures operational efficiency and system resilience but also offers scalability to accommodate the growing needs of the users. The Acceldata Environment, with its advanced architecture, empowers organizations to maintain a clear lens into their data landscape, fostering informed decision-making and proactive problem-solving.
ADOC Control Plane Components
Component - Control Plane
Infrastructure
Component - Infrastructure
| Name | Description |
|---|---|
| Reverse Proxy | Acts as an intermediary for requests from clients seeking resources from the ADOC platform. It ensures load balancing, provides SSL termination, and facilitates secure access control, enhancing security and system performance. |
| Dashplots | It is a visualization tool that offers real time data representations, helping users to interpret complex datasets through dashboards. It streamlines decision-making processes by transforming data into accessible and actionable insights. |
| Admin Central | This is the operational hub where administrators can perform a wide array of tasks, ranging from user management to system-wide configurations. It provides tools and interfaces to maintain and optimize the ADOC platform's performance. |
| Incident Manager | This component acts as a command center for incident detection and response. It automates the identification of system incidents, streamlines the response process, and ensures that incidents are resolved efficiently and effectively. |
| Alerts | The alert system is engineered to monitor system health and trigger notifications for predefined events or metrics that exceed acceptable thresholds. It helps maintain system integrity by prompting immediate attention to potential issues. |
| Management | The management component oversees operational processes and resources within the ADOC infrastructure. It is responsible for task scheduling, resource allocation, and operational oversight, ensuring the system runs at peak efficiency. |
Platform
Component - Platform
| Name | Description |
|---|---|
| Auth | This component is responsible for ensuring secure access to the ADOC platform. It handles authentication protocols, granting permissions and verifying the identities of users or services that interact with the platform. By employing robust security measures, such as two-factor authentication or OAuth tokens, the Auth module safeguards sensitive data and system integrity. |
| Secret Manager | Protects and manages sensitive information necessary for the platform's operations. It centralizes the handling of secrets, which is essential for maintaining secure and automated access to credentials and keys within the platform's ecosystem. |
| Columnar Database | Specialized database optimized for reading and analyzing data stored in columns, improving query performance and facilitating faster data analytics. |
| Cache | Serving as a temporary storage area, the Cache component dramatically boosts the ADOC platform's performance. By caching frequently accessed data, it minimizes latency and reduces the load on the primary storage system, ensuring that repeated data requests are served quickly and efficiently. This not only enhances user experience but also optimizes resource utilization. |
Compute
Component - Compute
| Name | Description |
|---|---|
| Databricks Connectors | Interfaces that link compute resources, ensuring seamless data flow and integration between compute operations and the ADOC platform. |
| Snowflake Connectors | Custom connectors designed to work with Snowflake data warehousing solutions, facilitating efficient data transfer and processing. |
| Consumer APP Worker | Manages the storage and analysis of time-series data, crucial for monitoring trends, patterns, and anomalies over time. |
Machine Learning
Component - ML
| Name | Description |
|---|---|
| Time Series ML | This component leverages advanced machine learning algorithms to perform in-depth analyses and generate predictions from time-series data. By examining patterns and trends over time, Time Series ML enhances the ADOC platform's capability to deliver forward-looking insights, predictive analytics, and intelligent forecasts that can inform decision-making and strategic planning. |
| Time Series Ingestion | The ingestion module is dedicated to the collection and assimilation of time-based data streams into the ADOC platform. It ensures that the data is accurately captured, timestamped, and made ready for further processing. This component is essential for handling high-velocity data that may come from various sources like IoT devices, application logs, or system monitors. |
| Time Series Management | This component serves as the administration hub for time-series data. It manages the lifecycle of the data, including storage, retention policies, indexing, and aggregation. Time Series Management is critical for maintaining data integrity, facilitating rapid query responses, and optimizing the storage system for efficient retrieval and analysis of time-dependent data. |
Reliability
Component - Reliability
| Name | Description |
|---|---|
| Pipeline Service | Monitors and ensures the operational health of services within the ADOC ecosystem, proactively addressing issues to maintain system reliability. |
| Reporting Service | Generates comprehensive reports on system performance, data quality, and operational insights, aiding in decision-making and strategy formulation. |
| Catalog Server | Manages and catalogs metadata from various data sources, providing a structured and searchable repository for efficient data retrieval and management. |
Data Plane
The Data Plane within the ADOC architecture is the layer that directly interacts with and manages the client's data resources. Managed by the client, it is structured to provide efficient data handling, processing, and storage capabilities, vital for comprehensive data observability. The Data Plane architecture employs SSL encryption to ensure the secure transmission of data, showcasing the commitment of the platform to data security and integrity.
Data Plane
Data Plane Services
Data Plane - Services
| Name | Description |
|---|---|
| Crawler | Discovers and catalogs available data sources and metadata within your environment, ensuring that all client datasets are known. |
| Analysis Services | Executes local data validation, transformation and computation of observability metrics (e.g. freshness, reliability). |
| Monitor Services | Continuously collects telemetry, health-checks and metrics from data sources and processing jobs for real-time observability. |
| Reverse Proxy | Provides a secure entry point for Control-Plane communication and client queries, routing traffic over an SSL-encrypted channel. |
Spark Services
Data Plane - Spark Services
| Name | Description |
|---|---|
| Spark Applications | Dedicated computational units that perform big data processing tasks using Apache Spark, which can handle batch and stream processing. |
| Profiler Jobs | These jobs analyze data performance and quality, profiling data to ensure it meets the standards and requirements of the organization. |
| Spark Operator | Orchestrates and manages Spark applications, ensuring efficient resource allocation and scheduling of data processing tasks. |
| YARN Scheduler | Manages and schedules jobs across the cluster, optimizing the use of system resources and improving job execution times. |
Client Data Sources
The foundation upon which data observability is built, comprising databases and storage systems that store and serve the raw data. Each data source is a repository that may include relational databases, NoSQL databases, file systems, and data warehouses.
These client-managed data sources are crucial for the ongoing collection and provision of data, enabling the Data Plane to perform its monitoring and analysis functions effectively.
All customer databases, data services, and cloud data repositories are included in the customer data section. Once the database is integrated into ADOC as a data source, ADOC continuously monitors the data on the customer data platform. Once configured, the data plane connects to your data platform to retrieve data for computation and reliability activities. Overall, ADOC's customer data sector is critical to its operations since it gives access to valuable client databases and data services. ADOC can assure dependable access to this data for multiple activities by continuously monitoring and configuring the data plane.
You can also create Pipelines to automate tasks. A Pipeline is a set of jobs that are executed in a sequential order. When a pipeline execution begins, all the jobs in the pipeline are executed. You can also use Acceldata's Software Development Kit (SDK) to monitor Python data pipelines.
Integration with the Control Plane
While the Data Plane focuses on direct data management and operational tasks, it integrates seamlessly with the Control Plane, which provides overarching governance, administration, and strategic management of the ADOC platform. This integrated approach ensures that data flows smoothly from the point of collection and storage to analysis, enabling enterprises to have a real-time and historical view of their data landscape.
Data communication between the Acceldata Environment and Customer Environment is carried through a secure channel.