Configure Pulse to Monitor Standalone Spark

This document provides you a step by step process on how to install single Pulse instance for multiple Spark Standalone clusters.

Pre-requisites

Ensure the following are present:

  1. Spark hosts: Refer to steps 1 and 2 mentioned below the note.
  2. Zookeeper hosts files: Refer to step 3 mentioned below the note.
  3. Log locations
  4. Spark history server locations
  5. Certificates (if any for Spark history server)
  6. Docker version

Prerequisites for enabling (TLS) HTTPS for Pulse Web UI Configuration using:

  1. Certificate File: cert.crt
  2. Certificate Key: cert.key
  3. CA Certificate: ca.crt (optional)
  4. Decide whether to keep the HTTP port (Default: 4000) open or not
  5. Decide on which port to use (default: 443)
  1. Obtain the fully qualified domain names (FQDN) for the Spark Master URLs for both clusters and include them in the spark_<clustername>.hosts file. The Spark hosts file should be structured as follows:
Bash
Copy
  1. Retrieve the fully qualified domain names (FQDN) for the Spark History Server URLs for both clusters. When requested, provide the URL in the following format:
Bash
Copy
  1. Obtain the fully qualified domain names (FQDN) for the Zookeeper Server URLs for both clusters and place them in the zk_<clustername>.hosts file. The Zookeeper Hosts file should adhere to the following format:
Bash
Copy
  1. Retrieve the log locations for the application and deployment logs, as well as the SPARK_HOME directory for both clusters.
  2. Ensure that the Docker version is >= 20.10.x.

Uninstallation

To uninstall agents, perform the following:

  1. To uninstall agents, you must run the hystaller uninstallcommand through their ansible setup.
  2. You must remove the Pulse Spark Hook Jars from the locations along with the related configurations from the Spark master and worker nodes.
  3. Acceldata team must then perform the following steps using the command below to backup and uninstall the existing Pulse application.
    1. Create a backup directory: mkdir -p /data01/backup
    2. As a backup, copy the entire config and work directories: cp -R $AcceloHome/config /data01/backup/cp -R $AcceloHome/work /data01/backup/``
    3. Uninstall the existing Pulse setup by running the following command: accelo uninstall local

OUTPUT

Bash
Copy

Executing this action will remove all files, folders, docker containers, docker images, and the entire Acceldata directory.

  1. Logout of the terminal session.

Download and Load Binaries and Docker Images

To download and load binaries and Docker images, perform the following:

When downloading the Pulse all-in-one TAR file, extract the hystaller binary directly from the package.
  1. Download the jars, hystaller, accelo binaries, and docker images from the download links provided by the Acceldata team.
  2. Move the Docker images and jars into the following directory:
Bash
Copy
  1. Copy the Binaries and Tar files into the /data01/images folder.
Bash
Copy
  1. Change the directory
Bash
Copy
  1. Extract the single tar file
Bash
Copy

OUTPUT

Bash
Copy
  1. To load the Docker images, execute the following command:
Bash
Copy
  1. Check if all the images are loaded to the server using the following command:
Bash
Copy

Replace 4.x.x with the Pulse version you want to install.

Configure the Cluster

To configure the cluster in Pulse, perform the following:

  1. Validate all the host files.
  2. Create the acceldata directory by running the following command:
Bash
Copy
  1. Place the accelo binary in this /data01/acceldata directory:
Bash
Copy
  1. Rename the accelo.linux binary to accelo .
Bash
Copy
  1. Change the directory:
Bash
Copy
  1. Run the following command to perform accelo init:
Bash
Copy
  1. Enter appropriate answers when prompted.
  2. When the Spark master is available, you can add the following parameter in the /etc/profile.d/ad.sh file to sync the Spark worker list from the Spark master URL.
Bash
Copy
  1. Run the following command to source the ad.sh file:
Bash
Copy
  1. Run the init command to provide the Pulse version:
Bash
Copy

OUTPUT

Bash
Copy

Replace 4.x.x with the Pulse version you want to install.

  1. Run accelo info command as follows:
Bash
Copy

OUTPUT

Bash
Copy

Replace 4.x.x with the Pulse version you want to install.

  1. To configure the cluster in Pulse, run the config cluster command:
Bash
Copy
  1. Provide the correct information when prompted. The output must appear as follows:
Bash
Copy
  1. Run the config cluster command for all the clusters and provide the appropriate answers when prompted.
Bash
Copy
  1. Run the config cluster command for Nifi Stand-Alone and select standalone > nifi.
Bash
Copy

Copy the License

Place the license file provided by the Acceldata team in the work directory as shown below:

Bash
Copy

Deploy Pulse Core Components

Deploy the Pulse core components by running the following command:

Bash
Copy

The output must appear as follows:

Bash
Copy

Deploy Add-ons

To deploy the Pulse add-ons, run the code below and select the required components for Spark standalone:

Bash
Copy

The output must appear as follows:

Bash
Copy

Configure Alerts Notifications

To configure alerts notifications, perform the following:

  1. Set the active cluster by running the following command:
Bash
Copy
  1. Configure the alerts notifications using the following command:
Bash
Copy

OUTPUT

Bash
Copy
  1. Set cluster2 as the active cluster:
Bash
Copy
  1. Configure the alerts for second cluster:
Bash
Copy
  1. Set cluster3 as the active cluster:
Bash
Copy
  1. Configure the alerts for the third cluster:
Bash
Copy
  1. Restart the alerts notifications:
Bash
Copy

OUTPUT

Bash
Copy

Database Push Configuration

Run the following command to push config to db:

Bash
Copy

Configure the Override

  1. Change the dir to work/<clustername> .
Bash
Copy
  1. Modify the override.yml file.
Bash
Copy
  1. Paste the below config in the file.
Bash
Copy

Do the above steps for all clusters.

Deploy the Pulse Agents

Install the new Pulse version x.x.x agents on all cluster nodes. Make a copy of the new hystaller file to /tmp or any executable location on all cluster nodes and then run the following command on all cluster nodes.

Change the following code snippet as per your environment
Bash
Copy

Reconfig Cluster

  1. After completing the edits to the override files as outlined above, the next step is to run the following command:
Bash
Copy

OUTPUT

Bash
Copy
  1. DB Push Config
Bash
Copy

Adding Edge Nodes for Monitoring

These are edge nodes that are not the part of the spark standalone cluster.
  1. Change the dir to work/<clustername> .
Bash
Copy
  1. Modify the hydra_hosts_override.yml file.
Bash
Copy
  1. Add the following code to add a host to a already existing host for pulse to monitor:
Bash
Copy
  1. Run the accelo reconfig cluster command for clusters with edge nodes that require monitoring by Pulse. Alternatively, for comprehensive coverage, perform a reconfig cluster on all clusters.
Bash
Copy
  1. Check the hydra_hosts.yml file which will now contain the new hosts as well. For example:
Bash
Copy

Configure Gauntlet

Updating the Gauntlet Crontab Duration

  1. Check if the ad-core.yml file is present or not by running the following command:
Bash
Copy
  1. If the file above is not present, then generate it by:
Bash
Copy
  1. Edit the ad-core.yml file

a. Open the file:

Bash
Copy

b. Update the CRON_TAB_DURATION env variable in the ad-gauntlet section:

Bash
Copy

This makes gauntlet run every 2 days at midnight.

c. The updated file will look something like this:

Bash
Copy

d. Save the file.

  1. Restart gauntlet service by running the command:
Bash
Copy

Updating the Gauntlet Dry Run Mode

  1. Check if the ad-core.yml file is present or not by running the following command:
Bash
Copy
  1. If the file above is not present, then generate it by:
Bash
Copy
  1. Edit the ad-core.yml file.

a. Open the file.

Bash
Copy

b. Update the DRY_RUN_ENABLE env variable in the ad-gauntlet section:

Bash
Copy

This will make the gauntlet delete the order elastic indices and mongo db data.

c. The updated file will look something like this:

Bash
Copy

d. Save the file.

  1. Restart gauntlet service by running the command:
Bash
Copy

Updating MongoDB Cleanup and Compaction Frequency in Hours

By default, when dry run is disabled MongoDB cleanup and compaction will run once a day. To configure the frequency, follow the steps listed below.

  1. Run the following command:
Bash
Copy
  1. Answer the following prompts, if you’re unsure about how many days you wish to retain. Then proceed with the default values.
Bash
Copy
  1. When the following prompt comes up, specify the hours of the day during which you would like MongoDB clean up and compaction to run. The value must be a CSV of hours as per the 24 hour time notation.
Bash
Copy
  1. Run the following command. When gauntlet runs the next time, MongoDB clean up and compaction will run at the specified hours, once per hour.
Bash
Copy

Enabling (TLS) HTTPS for Pulse Web UI Configuration

For details on the configuration, see Enable Native SSL/TLS for Pulse Web UI.

Set Up LDAP for Pulse UI

For details on the configuration, see Set up LDAP for Pulse UI.

Spark Jars Placements and Spark Config Changes

Perform the following steps for all the Spark Cluster Nodes:

  1. Add the following configuration in the metrics.properties file for Spark TimeSeries data:
Bash
Copy
  1. Add the following configuration in the spark-defaults.conf file for the Events data:
Bash
Copy
  1. Take the ad-spark-hook.jar file and put it in the following dir:
Bash
Copy
  1. Restart all the Spark services.

DotLog Download

We have introduced a feature that allows downloading of service logs in .log format. This file is not the original server log but an xlsx sheet merged into a .log format.

Perform the following to add a configurable parameter to enable this feature:

  1. Insert the dotLogFileDownload parameter into the feature flags property of the ad-graphql section found at the file path: $Acceldata_Home/config/docker/ad-core.yml.
Bash
Copy
  1. Restart the ad-graphl service using the following command:
Bash
Copy

Perform the following to enable new search options:

  1. Locate the “ad-graphql“ section in file $Acceldata_Home/config/docker/ad-core.yml and under the “environment“ key, add the following line:
Bash
Copy
  1. Restart the ad-graphl service using the following command:
Bash
Copy

Does a user in the Spark Standalone environment still see the Spark option in the left menu even after their access has been revoked from the role?

Create a different role that does not have Spark permission and assign that role to the user. Alternatively, you can leave it as is because even if the Spark entry is visible in the left navigation, the user will not be able to access it if access has been revoked from their role.

Are non-admin users in the Spark Standalone environment able to access Spark even though they have the appropriate role permissions for accessing Spark?

In the role edit window, click on "Select All" just below the Page permissions. Then, remove any permissions that you do not wish to grant and save the role. Any user assigned to this role should now have access to Spark in the Spark Standalone environment.

What is the reason for the absence of the Oozie workflow link between the Oozie workflow and the application ID in PULSE for a Spark job?

The Spark job's Application ID is generated by the Oozie service. It will only appear in the Pulse UI if it is available in Oozie's Web Service UI. If it is not present in the Oozie's Web Service UI, it will not be displayed in Pulse.
VariableType to search · ESC to discard
GlossaryType to search · ESC to discard
InsertType to search · ESC to discard
No matches
Next to read:
Set Up NiFi in Pulse

For additional help, contact www.acceldata.force.com OR call our service desk +1 844 9433282

Copyright © 2026

On This Page
Configure Pulse to Monitor Standalone SparkPre-requisitesUninstallationDownload and Load Binaries and Docker ImagesConfigure the ClusterCopy the LicenseDeploy Pulse Core ComponentsDeploy Add-onsConfigure Alerts NotificationsDatabase Push ConfigurationConfigure the OverrideDeploy the Pulse AgentsReconfig ClusterAdding Edge Nodes for MonitoringConfigure GauntletEnabling (TLS) HTTPS for Pulse Web UI ConfigurationSet Up LDAP for Pulse UISpark Jars Placements and Spark Config ChangesDotLog DownloadNew Search BarUpdating the Gauntlet Crontab DurationUpdating the Gauntlet Dry Run ModeUpdating MongoDB Cleanup and Compaction Frequency in Hours