FSImage Loading Takes More Time

Issue

FSImage processing in Pulse takes more time (for example, more than 10 hours) to complete for large FSImages (for example, ~40 GB).

Resolution

Update FS Analytics and Elasticsearch configuration

Apply the following configuration changes to improve the performance of FSImage parsing and Elasticsearch ingestion.

1. Step 1: Verify configuration files

On the Pulse node, check whether the following files exist at:

xxxxxxxxxx

/data01/acceldata/config/docker/addons

• ad-fsanalyticsv2-connector.yml
• ad-fs-elastic.yml

2. Step 2: Generate configuration files (if they do not already exist)

If the files do not exist, generate them using the following commands:

xxxxxxxxxx

accelo admin makeconfig ad-fsanalyticsv2-connector

accelo admin makeconfig ad-fs-elastic

3. Update environment variables

Open the configuration files and locate the environment section in each file.

• FS Analytics (ad-fsanalyticsv2-connector.yml)

Add the following entries:

xxxxxxxxxx

FSANALYTICS_ENABLE_OFFHEAP_MAP=true

FSANALYTICS_IMAGE_PARSING_BATCHSIZE=400000

ES_REFRESH_INTERVAL=1s

• Elasticsearch (ad-fs-elastic.yml)

Add the following entry:

xxxxxxxxxx

thread_pool.write.queue_size=10000

Result

These settings significantly improve FSImage parsing speed and Elasticsearch ingestion throughput, reducing overall FSImage processing time.