Configure Pulse to Monitor Standalone Spark

This document provides you a step by step process on how to install single Pulse instance for multiple Spark Standalone clusters.

Pre-requisites

Ensure the following are present:

Spark hosts: Refer to steps 1 and 2 mentioned below the note.
Zookeeper hosts files: Refer to step 3 mentioned below the note.
Log locations
Spark history server locations
Certificates (if any for Spark history server)
Docker version

Prerequisites for enabling (TLS) HTTPS for Pulse Web UI Configuration using ad-proxy:

Certificate File: cert.crt
Certificate Key: cert.key
CA Certificate: ca.crt (optional)
Decide whether to keep the HTTP port (Default: 4000) open or not
Decide on which port to use (default: 443)

Obtain the fully qualified domain names (FQDN) for the Spark Master URLs for both clusters and include them in the spark_<clustername>.hosts file. The Spark hosts file should be structured as follows:

Bash
    
xxxxxxxxxx
 
SparkMasterURLList:  - <http/s>://<Alias/FQDN of the Spark Master 1>:<Spark Master UI Port>  - <http/s>://<Alias/FQDN of the Spark Master 2>:<Spark Master UI Port>SparkWorkerURLList:  - <http/s>://<Alias/FQDN of the Spark Worker 1>:<Spark Worker Port>  - <http/s>://<Alias/FQDN of the Spark Worker 2>:<Spark Worker Port>
Copy

Retrieve the fully qualified domain names (FQDN) for the Spark History Server URLs for both clusters. When requested, provide the URL in the following format:

Bash
    
xxxxxxxxxx
 
<http/s>://<Alias/FQDN of the Spark History Server URL>:<Spark History Server URL>
Copy

Obtain the fully qualified domain names (FQDN) for the Zookeeper Server URLs for both clusters and place them in the zk_<clustername>.hosts file. The Zookeeper Hosts file should adhere to the following format:

Bash
    
xxxxxxxxxx
 
<http/s>://<Alias/FQDN for the Zookeeper Server>:<Zookeeper Server Port>
Copy

Retrieve the log locations for the application and deployment logs, as well as the SPARK_HOME directory for both clusters.
Ensure that the Docker version is >= 20.10.x.

Uninstallation

To uninstall agents, perform the following:

To uninstall agents, you must run the hystaller uninstallcommand through their ansible setup.
You must remove the Pulse Spark Hook Jars from the locations along with the related configurations from the Spark master and worker nodes.
Acceldata team must then perform the following steps using the command below to backup and uninstall the existing Pulse application.
1. Create a backup directory: mkdir -p /data01/backup
2. As a backup, copy the entire config and work directories: cp -R $AcceloHome/config /data01/backup/cp -R $AcceloHome/work /data01/backup/``
3. Uninstall the existing Pulse setup by running the following command: accelo uninstall local

OUTPUT

Bash
    
 
[root@nifihost1:data01 (ad-default)]$ accelo uninstall local✗ You're about to uninstall the local AccelData setup. This will also DELETE all persistent data from the current node. However, NONE of the remote no✔ You're about to uninstall the local AccelData setup. This will also DELETE all persistent data from the current node. However, NONE of the remote no✔ You're about to uninstall the local AccelData setup. This will also DELETE all persistent data from the current node. However, NONE of the remote noYou're about to uninstall the local AccelData setup. This will also DELETE all persistent data from the current node. However, NONE of the remote nodes will be affected. Please confirm your action [y/n]: : yWARN: Gauntlet is running in dry run mode. Disable this to delete indices from elastic and purge data from mongo DBUninstalling the AccelData components from local machine ...
Copy

Executing this action will remove all files, folders, docker containers, docker images, and the entire Acceldata directory.

Logout of the terminal session.

Download and Load Binaries and Docker Images

To download and load binaries and Docker images, perform the following:

When downloading the Pulse all-in-one TAR file, extract the hystaller binary directly from the package.

Download the jars, hystaller, accelo binaries, and docker images from the download links provided by the Acceldata team.
Move the Docker images and jars into the following directory:

Bash
    
xxxxxxxxxx
 
mkdir -p /data01/images
Copy

Copy the Binaries and Tar files into the /data01/images folder.

Bash
    
xxxxxxxxxx
 
cp </path/to/binaries/tar> /data01/images
Copy

Change the directory

Bash
    
xxxxxxxxxx
 
cd /data01/images
Copy

Extract the single tar file

Bash
    
xxxxxxxxxx
 
tar xvf <name_of_tar_file>.tar
Copy

OUTPUT

Bash
    
xxxxxxxxxx
 
[root@nifihost1 images]# tar xvf pulse-333-beta.tar./ad-alerts.tgz./ad-connectors.tgz./ad-dashplots.tgz./ad-database.tgz./ad-deployer.tgz./ad-director.tgz./ad-elastic.tgz./ad-events.tgz./ad-fsanalyticsv2-connector.tgz./ad-gauntlet.tgz./ad-graphql.tgz./ad-hydra.tgz./ad-impala-connector.tgz./ad-kafka-0-10-2-connector.tgz./ad-kafka-connector.tgz./ad-ldap.tgz./ad-logsearch-curator.tgz./ad-logstash.tgz./ad-notifications.tgz./ad-oozie-connector.tgz./ad-pg.tgz./ad-proxy.tgz./ad-pulsemon-ui.tgz./ad-recom.tgz./ad-sparkstats.tgz./ad-sql-analyser.tgz./ad-streaming.tgz./ad-vminsert.tgz./ad-vmselect.tgz./ad-vmstorage.tgz./accelo.linux./admon./hystaller
Copy

To load the Docker images, execute the following command:

Bash
    
xxxxxxxxxx
 
ls -1 *.tgz | xargs --no-run-if-empty -L 1 docker load -i
Copy

Check if all the images are loaded to the server using the following command:

Bash
    
xxxxxxxxxx
 
docker images | grep 4.x.x
Copy

Replace 4.x.x with the Pulse version you want to install.

Configure the Cluster

To configure the cluster in Pulse, perform the following:

Validate all the host files.
Create the acceldata directory by running the following command:

Bash
    
xxxxxxxxxx
 
cd /data01/mkdir -p acceldata
Copy

Place the accelo binary in this /data01/acceldata directory:

Bash
    
xxxxxxxxxx
 
cp </path/to/accelo/binary> /data01/acceldata
Copy

Rename the accelo.linux binary to accelo .

Bash
    
xxxxxxxxxx
 
mv /data01/acceldata/accelo.linux accelochmod +x /data01/acceldata/accelo
Copy

Change the directory:

Bash
    
xxxxxxxxxx
 
cd /data01/acceldata/accelo
Copy

Run the following command to perform accelo init:

Bash
    
xxxxxxxxxx
 
./accelo init
Copy

Enter appropriate answers when prompted.
When the Spark master is available, you can add the following parameter in the /etc/profile.d/ad.sh file to sync the Spark worker list from the Spark master URL.

Bash
    
xxxxxxxxxx
 
SYNC_SPARK_MASTER=true
Copy

Run the following command to source the ad.sh file:

Bash
    
xxxxxxxxxx
 
source /etc/profile.d/ad.sh
Copy

Run the init command to provide the Pulse version:

Bash
    
xxxxxxxxxx
 
accelo init
Copy

OUTPUT

Bash
    
xxxxxxxxxx
 
[root@nifihost1:~ (ad-default)]$ accelo initEnter the AccelData ImageTag: : 4.x.x✓ Done, AccelData Init Successful.
Copy

Replace 4.x.x with the Pulse version you want to install.

Run accelo info command as follows:

Bash
    
xxxxxxxxxx
 
accelo info
Copy

OUTPUT

Bash
    
xxxxxxxxxx
 
[root@nifihost1:~ (ad-default)]$ accelo infoWARN: Gauntlet is running in dry run mode. Disable this to delete indices from elastic and purge data from mongo DB      ___   ____________________    ____  ___  _________   /   | / ____/ ____/ ____/ /   / __ \/   |/_  __/   |  / /| |/ /   / /   / __/ / /   / / / / /| | / / / /| | / ___ / /___/ /___/ /___/ /___/ /_/ / ___ |/ / / ___ |/_/  |_\____/\____/_____/_____/_____/_/  |_/_/ /_/  |_| Accelo CLI Version:  4.x.xAccelo CLI Build Hash:  8ba4727f11e5b3f3902547585a37611b6ec74e7cAccelo CLI Build ID:  1700746329Accelo CLI Builder ID:  ZEdjMmxrYUdGdWRGOWhZMk5sYkdSaEVLCg==Accelo CLI Git Branch Hash:  TXdLaTlCVDFBdE56STNvPQo=AcceloHome:  /data01/acceldataAcceloStack:  ad-defaultAccelData Registry:  191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldataAccelData ImageTag:  4.x.xActive Cluster Name:  NotFoundAcceloConfig Mongo DB Retention days:  15AcceloConfig Mongo DB HDFS Reports Retention days:  15AccelConfig TSDB Retention days:  31dNumber of AccelData stacks found in this node:  0
Copy

Replace 4.x.x with the Pulse version you want to install.

To configure the cluster in Pulse, run the config cluster command:

Bash
    
xxxxxxxxxx
 
accelo config cluster
Copy

Provide the correct information when prompted. The output must appear as follows:

Bash
    
Copy

Run the config cluster command for all the clusters and provide the appropriate answers when prompted.

Bash
    
Copy

Run the config cluster command for Nifi Stand-Alone and select standalone > nifi.

Bash
    
Copy

Copy the License

Place the license file provided by the Acceldata team in the work directory as shown below:

Bash
    
Copy

Deploy Pulse Core Components

Deploy the Pulse core components by running the following command:

Bash
    
Copy

The output must appear as follows:

Bash
    
Copy

Deploy Add-ons

To deploy the Pulse add-ons, run the code below and select the required components for Spark standalone:

Bash
    
Copy

The output must appear as follows:

Bash
    
Copy

Configure Alerts Notifications

To configure alerts notifications, perform the following:

Set the active cluster by running the following command:

Bash
    
Copy

Configure the alerts notifications using the following command:

Bash
    
Copy

OUTPUT

Bash
    
Copy

Set cluster2 as the active cluster:

Bash
    
Copy

Configure the alerts for second cluster:

Bash
    
Copy

Set cluster3 as the active cluster:

Bash
    
Copy

Configure the alerts for the third cluster:

Bash
    
Copy

Restart the alerts notifications:

Bash
    
Copy

OUTPUT

Bash
    
Copy

Database Push Configuration

Run the following command to push config to db:

Bash
    
Copy

Install the new Pulse version x.x.x agents on all cluster nodes. Make a copy of the new hystaller file to /tmp or any executable location on all cluster nodes and then run the following command on all cluster nodes.

Change the following code snippet as per your environment

Bash
    
Copy

Reconfig Cluster

After completing the edits to the override files as outlined above, the next step is to run the following command:

Bash
    
Copy

OUTPUT

Bash
    
Copy

DB Push Config

Bash
    
Copy

Adding Edge Nodes for Monitoring

These are edge nodes that are not the part of the spark standalone cluster.

Change the dir to work/<clustername> .

Bash
    
Copy

Modify the hydra_hosts_override.yml file.

Bash
    
Copy

Add the following code to add a host to a already existing host for pulse to monitor:

Bash
    
Copy

Run the accelo reconfig cluster command for clusters with edge nodes that require monitoring by Pulse. Alternatively, for comprehensive coverage, perform a reconfig cluster on all clusters.

Bash
    
Copy

Check the hydra_hosts.yml file which will now contain the new hosts as well. For example:

Bash
    
Copy

Configure Gauntlet

Updating the Gauntlet Crontab Duration

Check if the ad-core.yml file is present or not by running the following command:

Bash
    
Copy

If the file above is not present, then generate it by:

Bash
    
Copy

Edit the ad-core.yml file

a. Open the file:

Bash
    
Copy

b. Update the CRON_TAB_DURATION env variable in the ad-gauntlet section:

Bash
    
Copy

This makes gauntlet run every 2 days at midnight.

c. The updated file will look something like this:

Bash
    
Copy

d. Save the file.

Restart gauntlet service by running the command:

Bash
    
Copy

Updating the Gauntlet Dry Run Mode

Check if the ad-core.yml file is present or not by running the following command:

Bash
    
Copy

If the file above is not present, then generate it by:

Bash
    
Copy

Edit the ad-core.yml file.

a. Open the file.

Bash
    
Copy

b. Update the DRY_RUN_ENABLE env variable in the ad-gauntlet section:

Bash
    
Copy

This will make the gauntlet delete the order elastic indices and mongo db data.

c. The updated file will look something like this:

Bash
    
Copy

d. Save the file.

Restart gauntlet service by running the command:

Bash
    
Copy

Updating MongoDB Cleanup and Compaction Frequency in Hours

By default, when dry run is disabled MongoDB cleanup and compaction will run once a day. To configure the frequency, follow the steps listed below.

Run the following command:

Bash
    
Copy

Answer the following prompts, if you’re unsure about how many days you wish to retain. Then proceed with the default values.

Bash
    
Copy

When the following prompt comes up, specify the hours of the day during which you would like MongoDB clean up and compaction to run. The value must be a CSV of hours as per the 24 hour time notation.

Bash
    
Copy

Run the following command. When gauntlet runs the next time, MongoDB clean up and compaction will run at the specified hours, once per hour.

Bash
    
Copy

Enabling (TLS) HTTPS for Pulse Web UI Configuration Using ad-proxy

Deployment and Configuration

Copy the cert.crt, cert.key and ca.crt (optional) files to $AcceloHome/config/proxy/certs location.
Check if ad-core.yml file is present or not.

Bash
    
Copy

If ad-core.yml file is not present, then generate the ad-core.yml file.

Bash
    
Copy

OUTPUT

Bash
    
Copy

Modify the ad-core.yml file.

a. Open the ad-core.yml file.

Bash
    
Copy

b. Remove the ports: field in the ad-graphql section of ad-core.yml .

Bash
    
Copy

c. The resulting ad-graphql section will look like this:

Bash
    
Copy

d. Save the file

Restart the ad-graphql container:

Bash
    
Copy

Check if the port is not exposed to the host.

Bash
    
Copy

Check if there are any errors in ad-graphql container:

Bash
    
Copy

Deploy the ad-proxy addons, run the following command, and select Proxy from the list and press enter.

Bash
    
Copy

OUTPUT

Bash
    
Copy

Check if there are any errors in the ad-proxy container:

Bash
    
Copy

Now you can access the Pulse UI using https://<pulse-server-hostname> . By default the port used is 443 .

Configuration

If you want to change the SSL port to another port, follow the below steps:

Check if ad-proxy.yml file is present or not.

Bash
    
Copy

Generate the ad-proxy.yml file if its not present.

Bash
    
Copy

OUTPUT

Bash
    
Copy

Modify the ad-proxy.yml .

a. Open the ad-proxy.yml file.

Bash
    
Copy

b. Change the host port in the ports list to the desired port.

Bash
    
Copy

The final file will look like this, if the host port is 6003 :

Bash
    
Copy

c. Save the file.

Restart the ad-proxy container.

Bash
    
Copy

Check if there aren’t any errors.

Bash
    
Copy

Now you can access the Pulse UI using https://<pulse-server-hostname>:6003 .

Set Up LDAP for Pulse UI

Check if the ldap.conf is present or not.

Bash
    
Copy

Run the accelo config ldap command to generate the default ldap.conf if not present already.

Bash
    
Copy

OUTPUT

Bash
    
Copy

Edit the file in path $AcceloHome/config/ldap/ldap.conf .

Bash
    
Copy

Configure file for below properties:

LDAP FQDN : FQDN where LDAP server is running
- host = [FQDN]
If port 389 is being used then
- insecureNoSSL = true
SSL root CA Certificate
- rootCA = [CERTIFICATE_FILE_PATH]
bindDN : to be used for ldap search need to be member of admin group
bindPW : <encrypted-password-string> for entering in database.
encryptedPassword = true , set this to true to enable the use of encrypted password.
baseDN used for user search
- Eg: (cn=users, cn=accounts, dc=accedata, dc=io)
Filter used for the user search
- Eg: (objectClass=person)
baseDN used for group search
- Eg: (cn= groups, cn=accounts, dc=acceldata, dc=io)
Group Search: Object class used for group search
- Eg: (objectClass= posixgroup)

Here is the command to check if a user has search entry access and group access in LDAP directory:

Bash
    
Copy

If the file is already generated, it will ask for the LDAP credentials to validate the connectivity and configurations, which are mentioned in the below steps.
Run the accelo config ldap command.

Bash
    
Copy

It will ask for the LDAP user credentials:

Bash
    
Copy

If things went correctly, the below confirmation message will be displayed:

Bash
    
Copy

Press ‘y' and press 'Enter’.

OUTPUT

Bash
    
Copy

Push the ldap config.

Bash
    
Copy

Run the deploy addon command.

Bash
    
Copy

Select the LDAP from the list shown and press 'Enter':

Bash
    
Copy

OUTPUT

Bash
    
Copy

Run the restart command.

Bash
    
Copy

Open Pulse Web UI and create default roles.
Add ops role with required access and all incoming users with ldap login will come under this role automatically.

Spark Jars Placements and Spark Config Changes

Perform the following steps for all the Spark Cluster Nodes:

Add the following configuration in the metrics.properties file for Spark TimeSeries data:

Bash
    
Copy

Add the following configuration in the spark-defaults.conf file for the Events data:

Bash
    
Copy

Take the ad-spark-hook.jar file and put it in the following dir:

Bash
    
Copy

Restart all the Spark services.

DotLog Download

We have introduced a feature that allows downloading of service logs in .log format. This file is not the original server log but an xlsx sheet merged into a .log format.

Perform the following to add a configurable parameter to enable this feature:

Insert the dotLogFileDownload parameter into the feature flags property of the ad-graphql section found at the file path: $Acceldata_Home/config/docker/ad-core.yml.

Bash
    
Copy

Restart the ad-graphl service using the following command:

Bash
    
Copy

New Search Bar

Perform the following to enable new search options:

Locate the “ad-graphql“ section in file $Acceldata_Home/config/docker/ad-core.yml and under the “environment“ key, add the following line:

Bash
    
Copy

Restart the ad-graphl service using the following command:

Bash
    
Copy

Does a user in the Spark Standalone environment still see the Spark option in the left menu even after their access has been revoked from the role?

Create a different role that does not have Spark permission and assign that role to the user. Alternatively, you can leave it as is because even if the Spark entry is visible in the left navigation, the user will not be able to access it if access has been revoked from their role.

Are non-admin users in the Spark Standalone environment able to access Spark even though they have the appropriate role permissions for accessing Spark?

In the role edit window, click on "Select All" just below the Page permissions. Then, remove any permissions that you do not wish to grant and save the role. Any user assigned to this role should now have access to Spark in the Spark Standalone environment.

What is the reason for the absence of the Oozie workflow link between the Oozie workflow and the application ID in PULSE for a Spark job?

The Spark job's Application ID is generated by the Oozie service. It will only appear in the Pulse UI if it is available in Oozie's Web Service UI. If it is not present in the Oozie's Web Service UI, it will not be displayed in Pulse.

Last updated on

Was this page helpful?

Configure Pulse to Monitor Standalone Spark

Pre-requisites

Uninstallation

Download and Load Binaries and Docker Images

Configure the Cluster

Copy the License

Deploy Pulse Core Components

Deploy Add-ons

Configure Alerts Notifications

Database Push Configuration

Configure the Override

Deploy the Pulse Agents

Reconfig Cluster

Adding Edge Nodes for Monitoring

Configure Gauntlet

Updating the Gauntlet Crontab Duration

Updating the Gauntlet Dry Run Mode

Updating MongoDB Cleanup and Compaction Frequency in Hours

Enabling (TLS) HTTPS for Pulse Web UI Configuration Using ad-proxy

Deployment and Configuration

Configuration

Set Up LDAP for Pulse UI

Spark Jars Placements and Spark Config Changes

DotLog Download

New Search Bar