Pulse Asynchronous Queue
The Pulse application monitors your big data environments. Big data applications send various events to Pulse continuously. Pulse needs to store these events, process them, and display the statistics from these events instantly on the respective Pulse dashboards.
Pulse uses an asynchronous boundary which is used as an asynchronous messaging queue. This messaging queue is managed by Neural Autonomic Transport System (NATS). The big data applications stream data directly to this messaging queue which acts as a broker. The storage system of this queue is robust enough to store raw data events for a long period of time.
Pulse deploys the asynchronous queue between various big data applications and the Pulse UI. The various applications monitored by Pulse, stream the metadata events directly to the asynchronous queue. The events are stored as raw data in the Queue. Once the data is received from an application (publisher), it can be consumed by various subscribers. Pulse UI is a subscriber which reads raw data (events) from the queue, processes it (Pulse uses the Mongo DB for storing processed data) and displays it on the respective dashboards.
The raw data in the asynchronous queue never changes. You can utilize the raw data in the Pulse queue for the following purposes.
- For debugging any errors.
- To go back in time and check logs.
- For Artificial Intelligence (AI) or Machine learning (ML) based applications.
The following image illustrated the architecture of Pulse Queue.
To view the data from the queue instantly, you must set the calendar filter to Last 15 minutes, on the required dashboard.
To learn more about how to enable the Pulse queue for a CDP deployment, see (Link Removed).