Upgrade to Version 4.1.3

You can follow the appropriate steps to upgrade Pulse to version 4.1.3 based on your current deployment:

• Upgrade Pulse from Version 4.1.x to 4.1.3

• Upgrade Pulse from Version 4.0.x to 4.1.3

• Upgrade Pulse from Version 3.8.x. to 4.1.3

Enable the runMigrations Property for Dashplots

Before performing a fresh installation or upgrading Pulse to a newer version, verify that the runMigrations property in the ad-dashplots-ui section of the ad-core configuration is set to true.

This property enables the required Dashplot database migrations. If it is not enabled during an upgrade, the ad-dashplots-ui service may fail to start and repeatedly restart with a database migration error similar to the following:

ERROR: column dashplot_reporting.cluster_key does not exist

For more information about the runMigrations property and other Dashplot configuration options, see Configure Dashplot Variables.

Upgrade Pulse from Version 4.1.x to 4.1.3

This section describes the required steps to upgrade from Pulse 4.1.x to 4.1.3.

1. Update ImageTag in:

Set it to:

2. Back up the existing accelo CLI file and download the latest accelo.linux.

3. Rename and make the CLI executable:

4. Push the updated config to the database:

5. For online installations, run the following command and press 'Y' at the prompt to get the most recent binaries.

6. For offline installations, download the latest image packs, then run the following command to upload them.

7. Delete the existing Impala Stream NATS Consumers (Required for 4.1.0 Upgrades Only).

   1. Delete all existing Impala Stream NATS consumers only when upgrading from Pulse 4.1.0 to Pulse 4.1.3

   2. This step is not required when upgrading from Pulse 4.1.2 to Pulse 4.1.3.

   3. For details, see the template below.

For example:

8. Delete the existing Pulse profiler stream.

9. Enable the ad-dashplots migration flags. For details, see Configure Dashplot Variables.

10. Stop the following containers, if they are present.

11. Delete the following directories, if they are present.

12. Remove the following containers, if they are present.

13. Run the following command to bring up the ad-pg / ad-pg-ui containers and create the required pg / pgui directory paths:

14. Restart all the containers so that they use the updated images.

15. Uninstall the Hydra agent.

16. Install the Hydra agent:

17. Reconfigure the cluster to get the updated configuration for agents.

18. To push the updated configuration to the database, execute the following command

19. Confirm the ImageTag update:

20. Migrate the hooks to the relevant supported versions. For details, see Hook Migration.

21. [Optional] If you use custom dashboards, import them into Dashplots. For more information, see Dashboard.

22. If Pulse monitors multiple clusters, repeat the multi-cluster procedure for each remaining cluster. Skip the current active cluster, as it has already been updated. For more information, see Pulse Monitor Multi-Cluster Deployment.

Upgrade Pulse from Version 4.0.x to 4.1.3

This section describes the required steps to upgrade from Pulse 4.0.x to 4.1.3.

Step 1: Update Container Runtime Configuration

Update the /etc/profile.d/ad.sh file based on your container runtime.

For Podman:

For Docker:

Once modified, run the following command to apply the updated environment variables.

Step 2: Uninstall Hydra Agent

Hydra agents must be uninstalled to stop NATS stream data from being sent.

SSH-based deployment

Mpack and parcel-based deployments

• Uninstall Hydra agents from the CDP or Ambari UI.

• This step is required to stop NATS streams from being sent from the agent's side. Before doing further steps, ensure all NATS stream data has been processed before continuing.

Step 3: Update Accelo CLI

1. Take a backup of the existing accelo CLI binary.

2. Download the latest accelo.linux CLI.

3. Replace the existing binary and make it executable:

Step 4: Elasticsearch Migration: ES 8.16.6 → ES 8.19.0

Before upgrading Pulse from 4.0.x to 4.1.2 or later, both Elasticsearch instances must be upgraded from 8.16.6 to 8.19.0:

• ad-elastic

• ad-fs-elastic

For details about migrating the Elasticsearch DB from 8.16.6 → 8.19.0, see Upgrade ElasticSearch from ES 8.16.6 to ES 8.19.0.

Step 5: Update the Image Tag

Update the ImageTag in $AcceloHome/config/accelo.yml:

Set the version to:

Push the updated configuration to the database:

Verify the update:

Step 6: Offline Installations: Load Latest Images

For offline installations, download the latest image packs and load them:

Step 7: Delete the existing Pulse profiler stream.

Step 8: Run Migration

Run the migration command from $AcceloHome.

For example:

• -v: specifies the version you are migrating to (4.1.0). Note: Keep the version as -v 4.1.0, even when you are migrating to any update release version in 4.1.x.

• -i: specifies the version you are migrating from (4.0.0).

To migrate from version 4.0.0 to 4.1.3, the migration command looks as shown below.

Make sure you run the following command from $AcceloHome.

For a single cluster, run:

For multi-cluster setups, run:

If any issues occur during migration, rerun the same command.

Step 9: Online Installations: Pull Latest Images

For online installations, run:

Press Y when prompted to download the most recent binaries.

Step 10: Stop the following containers, if they are present.

Step 11: Delete the following directories, if they are present.

Step 12: Remove the following containers, if they are present.

Step 13: Run the following command to bring up the ad-pg / ad-pg-ui containers and create the required pg / pgui directory paths.

Step 14: Restart all the containers so that they use the updated images.

Step 15: (Optional) Migrate from old Actions (ad-director) to the new Actions service.

Pulse does not support Kerberos migration from old actions to new actions. For new actions, the Kerberos plugin had to be added before running any action plugin if the cluster was kerberized.

1. If the ad-director was previously configured, verify that the new Actions Service (ad-axnserver) container is running.

2. If the Actions Service container is not present, deploy it using the command:

3. When prompted, select Actions Service.

4. Press Enter to start deployment.

Verify the deployment

After the deployment completes, confirm that the container is running:

Expected output (status should be Up):

Check the container logs to confirm that the service has started successfully:

Reconfigure Notifications

If notifications (Email, Slack, Jira) were configured in ad-director, enable them again for the new Actions service.

Example:

After configuration completes:

• The notifications configuration file is generated.

• The configuration is pushed to the Pulse database.

Run the migration

Once the Actions Service is running and healthy, run:

Step 16: Install Hydra Agent

1. Deploy Hydra agents:

For mpack- or parcel-based deployments, install the Hydra agent using Ambari or Cloudera Manager.

2. Reconfigure the cluster to apply the updated agent configuration:

3. Push the updated configuration to the database:

Step 16: (Optional) Enable native SSL in Pulse UI

• If SSL was previously enabled using ad-proxy, enable native SSL support in Pulse UI.

• The ad-proxy service has been removed starting in Pulse 4.1.x. Native SSL must now be configured directly in Pulse UI.

• For instructions, see Enable Native SSL/TLS for Pulse Web UI.

Step 17: Migrate the hooks to the relevant supported versions. For details, see Hook Migration.

Step 18 (Optional): If you use custom dashboards, import them into Dashplots. For more information, see Dashboard.

Step 18 (Optional): If Pulse monitors multiple clusters, repeat the multi-cluster procedure for each remaining cluster. Skip the current active cluster, as it has already been updated. For more information, see Pulse Monitor Multi-Cluster Deployment.

Step 19: Configure Retention

To configure MongoDB cleanup and compaction scheduling:

Step 20: Create Database Indices

Run the following command to create indices for database collections:

If readOnlyRootFSEnabled is set to true in $AcceloHome/config/accelo.yml, follow these steps:

1. Open $AcceloHome/config/accelo.yml and set:

2. Run:

3. Reset readOnlyRootFSEnabled to true and rerun:

Upgrade Pulse from Version 3.8.x. to 4.1.3

Step 1: Update Container Runtime Configuration

Update /etc/profile.d/ad.sh based on the container orchestration tool in use.

For Docker:

For Podman:

Once modified, run the following command to apply the updated environment variables.

Step 2: Uninstall Hydra Agent

Hydra agents must be uninstalled to stop NATS stream data from being sent.

SSH-based deployment

Mpack and parcel-based deployments

• Uninstall Hydra agents from the CDP or Ambari UI.

• This step is required to Stop nats streams data being sent from agents side. Ensure all NATS stream data has been processed before continuing.

Step 3: Update Accelo CLI

1. Take a backup of the existing Accelo binary.

2. Download the latest accelo.linux CLI.

3. Replace the binary and make it executable:

Step 4: Elasticsearch Migration: ES 8.16.6 → ES 8.19.0

Before upgrading Pulse from 3.8.x to 4.1.2 or later, both Elasticsearch instances must be upgraded from 8.16.6 to 8.19.0:

• ad-elastic

• ad-fs-elastic

For details about migrating the Elasticsearch DB from 8.16.6 → 8.19.0, see Upgrade ElasticSearch from ES 8.16.6 to ES 8.19.0.

Step 5: Update ImageTag

Update the ImageTag in:

Set:

Push the updated configuration to the database:

Verify the update:

Step 6: Offline Installations: Load Latest Images

For offline installations, download the latest image packs and load them:

Step 7: Run Migration

For example:

• -v: specifies the version you are migrating to (4.1.0). Note: Keep the version as -v 4.1.0, even when you are migrating to any update release version in 4.1.x.

• -i: specifies the version you are migrating from (3.8.0).

To migrate from version 3.8.0 to 4.1.3, the migration command looks as shown below.

Make sure you run the following command from $AcceloHome

• For a single cluster, run:

• For multi-cluster setups, run:

If any issues occur, rerun the same migration command.

Step 8: Deploy Core Services

After migration, deploy core services:

Step 9: Online Installations: Pull Latest Images

For online installations:

Press Y when prompted to download the latest binaries.

Step 10: Stop the following containers, if they are present.

Step 11: Delete the following directories, if they are present.

Step 12: Remove the following containers, if they are present.

Step 13: Run the following command to bring up the ad-pg / ad-pg-ui containers and create the required pg / pgui directory paths.

Step 14: Restart all containers to apply the updated images.

Step 15: (Optional) Migrate from old Actions (ad-director) to the new Actions service

1. If the ad-director was previously configured, verify that the new Actions Service (ad-axnserver) container is running.

2. If the Actions Service container is not present, deploy it using the command:

3. When prompted, select Actions Service.

4. Press Enter to start deployment.

Verify the deployment

After deployment completes, confirm that the container is running:

Expected output (status should be Up):

Check the container logs to confirm that the service has started successfully:

Reconfigure Notifications

If notifications (Email, Slack, Jira) were configured in ad-director, enable them again for the new Actions service.

Example:

After configuration completes:

• The notifications configuration file is generated.

• The configuration is pushed to the Pulse database.

Run the migration

Once the Actions Service is running and healthy, run:

Step 16: Install Hydra Agent

Deploy Hydra agents:

Reconfigure the cluster:

Push updated configuration to the database:

Step 17: (Optional) Enable native SSL in Pulse UI

• If SSL was previously enabled using ad-proxy, enable native SSL support in Pulse UI.

• The ad-proxy service has been removed starting in Pulse 4.1.x. Native SSL must now be configured directly in Pulse UI.

• For instructions, see Enable Native SSL/TLS for Pulse Web UI.

Step 18: Migrate the hooks to the relevant supported versions. For details, see Hook Migration.

Step 19 (Optional): If you use custom dashboards, import them into Dashplots. For more information, see Dashboard.

Step 20 (Optional): If Pulse monitors multiple clusters, repeat the multi-cluster procedure for each remaining cluster. Skip the current active cluster, as it has already been updated. For more information, see Pulse Monitor Multi-Cluster Deployment.

Step 21: Configure Retention

To configure MongoDB cleanup and compaction:

Step 22: Create Database Indexes

Create database indexes:

If readOnlyRootFSEnabled Is Set to true

If readOnlyRootFSEnabled is enabled in $AcceloHome/config/accelo.yml, perform the following steps:

1. Set readOnlyRootFSEnabled to false.

2. Push configuration and restart:

3. Create indexes:

4. Set readOnlyRootFSEnabled back to true.

5. Push configuration and restart again:

Hook Migration

After upgrading Pulse to a newer Pulse version, migrate your existing Hive, Tez, and Spark hooks to the supported hook version required for that release to ensure continued compatibility and event collection.

For information about the supported hook version for each Pulse release, see Supported Hook Versions.

For details about the hook migration steps, see:

• For Hive and Tez

  • Configure ODP Hive and Tez

  • Configure HDP Hive, Tez, and Sqoop

  • Configure CDP Hive and Tez

  • Configure CDH Hive and Sqoop

• For Spark

  • Configure ODP Spark 2 and 3

  • Configure HDP Spark

  • Configure CDP Spark

  • Configure CDH Spark