Enable NATS (ad-events) in Cluster Mode
This document explains the steps to enable NATS (ad-events service) in a Pulse for a multi-node setup. This setup requires you to have at least three nodes in your cluster. One of the three nodes must be the Pulse core node.
NATS (ad-events) provides messaging and event streaming for Pulse services. You can configure NATS in cluster mode to distribute the messaging workload across multiple nodes.
This guide describes how to migrate an existing single-node NATS deployment to a multi-node NATS cluster in Pulse.
Prerequisites
Before you begin, ensure that:
- You have provisioned at least three nodes for the NATS cluster.
- The cluster contains an odd number of nodes, such as 3 or 5.
- Each node has a fully qualified domain name (FQDN) that the other nodes can resolve.
- You have access to the Pulse master node and standalone nodes.
- You have planned a maintenance window for the migration.
Important: Migrating from a single-node NATS deployment to a multi-node cluster requires temporarily stopping the ad-events service. The recovery time depends on the volume of data stored in NATS.
Step 1: Create the NATS Server Configuration
On each NATS host, create a nats-server.conf file in the $AcceloHome directory.
Add the following configuration:
debug: true
trace: false
server_name: ns1
port: 4222
http_port: 8222
jetstream = {
store_dir: "/data/nats-server/"
max_file: 500G
}
cluster = {
name: "JSC"
port: 6222
listen: "0.0.0.0:6222"
routes = [
nats://<hostname_FQDN2>:19008
nats://<hostname_FQDN3>:19008
]
}
Parameter | Description |
server_name | Specify a unique server name for each host, such as ns1, ns2, or ns3. |
cluster.routes | Specify the FQDNs of all other NATS hosts, excluding the current host. |
For example, when configuring ns1, include the routes for ns2 and ns3.
Note: Keep the configuration file in $AcceloHome temporarily. You will move it to the required directory during migration.
Step 2: Migrate to a Multi-Node NATS Cluster
2.1 Move the configuration files
On each NATS host, move the nats-server.conf file to:
$AcceloHome/config/db/nats-server.conf
2.2 Generate the service configuration files
On the Pulse master node, run:
accelo admin makeconfig ad-core
$AcceloHome/config/docker/ad-core.yml
accelo admin makeconfig ad-sa-events
$AcceloHome/config/docker/addons/ad-sa-events.yml
2.3 Stop the NATS service
On the Pulse master node, stop the running ad-events container:
docker stop ad-events_default
2.4 Remove the NATS metadata directory
On the Pulse master node, run:
rm -rf "$AcceloHome/data/nats/\$SYS"
2.5 Configure host resolution (optional)
If a DNS resolver is unavailable, ensure that /etc/hosts contains entries for all NATS hosts.
On the Pulse master node, open:
vim $AcceloHome/config/docker/ad-core.yml
ad-events:
volumes:
- /etc/hosts:/etc/hosts
vim $AcceloHome/config/docker/addons/ad-sa-events.yml
ad-sa-events:
volumes:
- /etc/hosts:/etc/hosts
2.6 Start the NATS service
On the Pulse master node, run:
docker start ad-events_default
2.7 Deploy NATS on standalone nodes
On each standalone Pulse node, run:
accelo deploy addons
Complete the deployment by following the prompts.
Step 3: Update NATS Endpoints in Pulse
For each cluster configured in Pulse, open the corresponding configuration file:
$AcceloHome/config/acceldata_<clusterName>.conf
nats.server.host.list="hostname_FQDN1:19009,hostname_FQDN2:19009,hostname_FQDN3:19009"
Push the updated configurations to the database:
accelo admin database push-config -a
Step 4: Update NATS Configuration for Pulse Microservices
If any of the following microservices are deployed, update their NATS connection settings:
- ad-pulse-profiler
- ad-sparkstats
- ad-yarn-optimizer
- ad-alfred
- ad-druid-analyzer
- ad-axnserver
For each deployed service, generate the configuration file:
accelo admin makeconfig <serviceName>
accelo admin makeconfig ad-yarn-optimizer
$AcceloHome/config/docker/addons/
After updating the required microservice configurations, restart the Pulse services:
accelo restart all
Step 5: Update the NATS Configuration for Pulse Agents
For each cluster configured in Pulse, open the override.yml file:
$AcceloHome/work/<clusterName>/override.yml
base:
nats_endpoint: hostname_FQDN1:19009,hostname_FQDN2:19009,hostname_FQDN3:19009
agent:
impala_stream_replica: "3"
hbazer_stream_replica: "3"
components:
impala_agent_objectstore_replication: "3"
Parameter | Description |
base.nats_endpoint | Specify the FQDNs and ports of all NATS hosts, separated by commas. |
agent.impala_stream_replica | Set the value to the number of NATS hosts. |
agent.hbazer_stream_replica | Set the value to the number of NATS hosts. |
components.impala_agent_objectstore_replication | Set the value to the number of NATS hosts. |
For example, if your NATS cluster contains three hosts, set the replication values to 3.
Save the configuration file and run:
accelo reconfig cluster -f
Note: Repeat this procedure for every cluster configured in Pulse.
Step 6: Update NATS Endpoints for Integration Hooks
If you have configured integration hooks for Hive, Tez, Spark, Trino, or Airflow, update their NATS endpoints to include all NATS hosts.
Hive and Tez Hooks
- Open the Hive and Tez hook configurations.
- Update the following property:
ad.events.streaming.servers=hostname_FQDN1:19009,hostname_FQDN2:19009,hostname_FQDN3:19009
- Save the changes.
- Restart the Hive and Tez services.
Spark Hook
- Open the Spark configuration in the appropriate location:
- For standalone Spark deployments, update the custom spark-defaults configuration.
- For managed Spark deployments, update the configuration through Cloudera Manager or Ambari.
- Update the following property:
spark.ad.nats.server.host.list=http://hostname_FQDN1:19009,http://hostname_FQDN2:19009,http://hostname_FQDN3:19009
- Save the changes.
Trino Hook
- Open the following configuration file:
/usr/odp/current/trino/conf/event-listener.properties
- Update the following property:
ad.events.streaming.servers=hostname_FQDN1:19009,hostname_FQDN2:19009,hostname_FQDN3:19009
- Save the changes.
- Restart the Trino coordinator.
Airflow Hook
- Locate the config.ini file associated with the Airflow listener under:
/usr/local/airflow/plugins
- Update the following property:
ad.events.url=hostname_FQDN1:19009,hostname_FQDN2:19009,hostname_FQDN3:19009
- Save the changes.
Step 7: Verify the NATS Cluster Configuration
After completing the migration, verify that:
- The NATS containers are running on the master and standalone nodes.
- All configured NATS hosts can communicate with one another.
- The NATS cluster recognizes the expected number of nodes.
- Pulse microservices can connect to the updated NATS endpoints.
- Pulse agents use the updated NATS configuration.
- Configured integration hooks can send events to NATS.

Have a suggestion?