Upgrade ElasticSearch from ES 8.16.6 to ES 8.19.0

Before upgrading Pulse from version 3.8.x to 4.1.x, you must upgrade both Elasticsearch instances from ES 8.16.6 / ES 8.18.4 to ES 8.19.0:

  • ad-elastic

  • ad-fs-elastic

This step is mandatory and must be completed before running the main Pulse migration command.

Ensure the Elasticsearch cluster health is yellow or green before proceeding with the Elasticsearch migration command.


How the Migration Flow Works

The migration command performs the following steps internally:

1. Pre-migration Cluster Health Check

The command checks the Elasticsearch cluster health before starting the migration.

  • If the cluster status is red, the migration will not proceed.

  • The command retries the health check until the cluster status becomes yellow or green.

Default retry configuration:

  • Maximum retries: 20

  • Retry interval: 1m

If all retry attempts are exhausted and the cluster is still red, the migration fails.

2. Migration Execution

Once the cluster health is yellow or green, the Elasticsearch version upgrade is performed.

3. Post-migration Sleep

After the migration completes, the system waits for 2m to allow the Elasticsearch container to stabilize.

4. Post-migration Cluster Health Check

After the sleep interval, the command performs another cluster health check using the same retry configuration.

If the post-migration health check fails after all retry attempts, verify the cluster health manually using the commands provided in the Manual Health Check section.


Accelo Elasticsearch Migration Command Flags

accelo admin [component] migrate [flags]

Supported components:

ad-elastic ad-fs-elastic

Flag

Short

Description

Default

--host

-d

Hostname where Elasticsearch is hosted

localhost

--port

-p

Port to which the Elasticsearch HTTP API is bound

19013

--username

-u

Elasticsearch username

pulse

--password

-e

Elasticsearch password encrypted using accelo admin encrypt

--

--online

-o

Pull Elasticsearch migration images from ECR

false

--retries

-r

Maximum number of health check retries

20

--retry-interval

-i

Interval between health check retries

1m

--sleep

-s

Time to wait after the Elasticsearch container becomes available

2m


ad-elastic Migration Procedure

Online Installation

Run the following command.

accelo admin ad-elastic migrate --online

Example:

$ accelo admin ad-elastic migrate --online INFO: Running ad-elastic migrate command Is the 'ad-elastic' service up and running? [y/n]: : y INFO: sleep duration: 2m0s, max retries: 20, retry interval: 1m0s Now, trying to logon to the AccelData registry... Docker login succeeded ✓ INFO: Current ad-elastic version from container label: 4.0.0 INFO: Performing cluster health check before migration to ES 8.19.0 (Current ES: 8.18.4)... INFO: Cluster health status: green INFO: Cluster health check passed INFO: Pulling migrate-8.19.0 image INFO: migrate-8.19.0 pull successful INFO: Tagging migrate-8.19.0 image to pulse version INFO: Removing old ad-elastic container (4.0.0 / ES 8.18.4) INFO: Creating new ad-elastic container with ES 8.19.0 INFO: Waiting for Elasticsearch to start... INFO: Performing cluster health check after migration to ES 8.19.0... INFO: Cluster health status: green INFO: completed:(ES 8.18.4) -> (ES 8.19.0)

Offline Installation

Load the migration image:

$ docker load -i ad-elastic8.tgz Loaded image: 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-elastic:migrate-8.19.0

Verify the image:

docker images | grep elastic

Run the migration command:

$ accelo admin ad-elastic migrate INFO: Running ad-elastic migrate command Is the 'ad-elastic' service up and running? [y/n]: : y INFO: sleep duration: 2m0s, max retries: 20, retry interval: 1m0s Now, trying to logon to the AccelData registry... Docker login succeeded ✓ INFO: Current ad-elastic version from container label: 4.0.0 INFO: Performing cluster health check before migration to ES 8.19.0 (Current ES: 8.18.4)... INFO: Cluster health status: green INFO: Cluster health check passed INFO: Pulling migrate-8.19.0 image INFO: migrate-8.19.0 pull successful INFO: Tagging migrate-8.19.0 image to pulse version INFO: Removing old ad-elastic container (4.0.0 / ES 8.18.4) INFO: Creating new ad-elastic container with ES 8.19.0 INFO: Waiting for Elasticsearch to start... INFO: Performing cluster health check after migration to ES 8.19.0... INFO: Cluster health status: green INFO: completed:(ES 8.18.4) -> (ES 8.19.0)

Verify the container:

docker ps -a | grep ad-elastic_default

ad-fs-elastic Migration Procedure

Online Installation ( (pulls images from ECR)

Run the following command.

accelo admin ad-fs-elastic migrate --online

Example:

[root@odppulse801:acceldata (ad-default)]$ accelo admin ad-fs-elastic migrate --online INFO: Running ad-fs-elastic migrate command Is the 'ad-fs-elastic' service up and running? [y/n]: : y ✔ Is the 'ad-fs-elastic' service up and running? [y/n]: : y Now, trying to logon to the AccelData registry... Docker login succeeded ✓ INFO: Current ad-fs-elastic version from container label: 4.0.0 INFO: Performing cluster health check before migration to ES 8.19.0 (Current ES: 8.18.4)... INFO: Cluster health status: green INFO: Cluster health check passed INFO: migrate-8.19.0 image already present locally INFO: Tagging migrate-8.19.0 image to pulse version INFO: Removing old ad-fs-elastic container (4.0.0 / ES 8.18.4) INFO: Creating new ad-fs-elastic container with ES 8.19.0 INFO: Waiting for Elasticsearch to start... INFO: Performing cluster health check after migration to ES 8.19.0... INFO: Cluster health status: green INFO: completed:(ES 8.18.4) -> (ES 8.19.0)

For Offline Installation (air-gapped):

Info

If the image 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-elastic:migrate-8.19.0 is already present locally, there is no need to load it again using docker load -i.

Download the required migration images in an air-gapped environment.

Load the migration image if it is not already present:

[root@odppulse801:acceldata (ad-default)]$ docker load -i ad-elastic8.tgz Loaded image: 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-elastic:migrate-8.19.0

Verify the image:

[root@odppulse801:acceldata (ad-default)]$ docker images | grep elastic 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-elastic

Run the migration command:

[root@odppulse801:acceldata (ad-default)]$ accelo admin ad-fs-elastic migrate INFO: Running ad-fs-elastic migrate command Is the 'ad-fs-elastic' service up and running? [y/n]: : y ✔ Is the 'ad-fs-elastic' service up and running? [y/n]: : y█ Now, trying to logon to the AccelData registry... Docker login succeeded ✓ INFO: Current ad-fs-elastic version from container label: 4.0.0 INFO: Performing cluster health check before migration to ES 8.19.0 (Current ES: 8.18.4)... INFO: Cluster health status: green INFO: Cluster health check passed INFO: migrate-8.19.0 image already present locally INFO: Tagging migrate-8.19.0 image to pulse version INFO: Removing old ad-fs-elastic container (4.0.0 / ES 8.18.4) INFO: Creating new ad-fs-elastic container with ES 8.19.0 INFO: Waiting for Elasticsearch to start... INFO: Performing cluster health check after migration to ES 8.19.0... INFO: Cluster health status: green INFO: completed:(ES 8.18.4) -> (ES 8.19.0)

Verify the container:

[root@odppulse801:acceldata (ad-default)]$ docker ps -a | grep ad-elastic_default bbfe825691ef 753de6514420 "/bin/tini -- /usr/l…" 10 minutes ago Up 10 minutes 0.0.0.0:19013->9200/tcp, 0.0.0.0:19014->9300/tcp ad-elastic_default

Manual Health Check

If the migration fails due to cluster health check timeout (cluster remains red), verify the cluster health manually:

ad-elastic Health Check

Default port: 19013

Command

accelo admin hc elastic cluster-health

Output

INFO: elastic cluster health { "active_primary_shards": 34, "active_shards": 34, "active_shards_percent_as_number": 100, "cluster_name": "elasticsearch", "delayed_unassigned_shards": 0, "initializing_shards": 0, "number_of_data_nodes": 1, "number_of_in_flight_fetch": 0, "number_of_nodes": 1, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_primary_shards": 0, "unassigned_shards": 0 }

ad-fs-elastic Health Check

Default port: 19038

Command

accelo admin hc elastic cluster-health -d localhost -p 19038

Output

INFO: elastic cluster health { "active_primary_shards": 0, "active_shards": 0, "active_shards_percent_as_number": 100, "cluster_name": "elasticsearch", "delayed_unassigned_shards": 0, "initializing_shards": 0, "number_of_data_nodes": 1, "number_of_in_flight_fetch": 0, "number_of_nodes": 1, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_primary_shards": 0, "unassigned_shards": 0 }