Upgrade Apache NiFi and NiFi Registry from ODP 3.3 to ODP 3.4

This page describes the upgrade and migration of Apache NiFi and NiFi Registry from ODP 3.3.6.5 to ODP 3.4.3.0. It covers pre-upgrade verification, configuration and data backups, service installation, data restoration, post-upgrade validation, and known issues identified during the upgrade.

The upgrade involves migrating NiFi flows, NiFi Registry metadata, versioned flows, and configuration settings from NiFi 1.28.1 to NiFi 2.8.0.

1. Upgrade Overview

The following table lists the source and target versions.

The upgrade verification was performed in a single-node environment.

2. Configure the Upgrade Environment

Before starting the upgrade, configure the Ambari connection parameters and create a backup directory.

1. Set the Ambari connection parameters.

export AMBARI_HOST=<IP_Address>
export AMBARI_PORT=8080
export AMBARI_USER=admin
export AMBARI_PASSWORD=admin

2. Retrieve the cluster name.

export CLUSTER_NAME=$(curl -s -u "$AMBARI_USER:$AMBARI_PASSWORD" \
 "http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters" \
 | grep -o '"cluster_name" : "[^"]*' \
 | cut -d'"' -f4)

3. Set the timestamp and backup directory.

export TS=$(date +%Y%m%d-%H%M%S)
export B=/var/tmp/odp-upgrade-backups-$TS

4. Example backup directory:

/var/tmp/odp-upgrade-backups-20261005-191342

3. Verify the Existing NiFi Environment

Before backing up the services, verify the existing NiFi flows, Registry buckets, and stored snapshots.

3.1 Verify NiFi Flow Components

Run the following command to identify the components in the existing NiFi flow configuration.

zcat /var/lib/nifi/conf/flow.json.gz \
  | grep -o '"componentType":"[A-Z_]*"' \
  | sort \
  | uniq -c

Expected output:

6 CONNECTION
1 FLOW_REGISTRY_CLIENT
6 PROCESS_GROUP
12 PROCESSOR

The process group count includes the root process group.

3.2 Verify NiFi Registry Buckets

Run the following command to retrieve the existing Registry buckets.

curl -s http://$(hostname -f):61080/nifi-registry-api/buckets

The verification identified the following bucket:

Bucket: odp_smoke_bucket_1791206172
ID: bb1fc589-5342-4646-a621-a3e9dd75b897

3.3 Verify Stored Flow Snapshots

Run the following command to count the existing flow snapshots.

find /var/lib/nifi-registry/flow_storage \
  -name '*.snapshot' | wc -l

Expected output:

6

3.4 Identify Existing Data Directories

The following table lists the source directories that contain NiFi and NiFi Registry configurations and data.

4. Stop Services and Back Up Data

Stop NiFi and NiFi Registry before creating backups to prevent data changes during the backup process.

4.1 Stop NiFi and NiFi Registry

1. Log in to the Ambari UI.
2. Stop the NIFI service.
3. Stop the NIFI_REGISTRY service.
4. Verify that the NiFi processes have stopped.

ps aux | grep -i 'org.apache.nifi' \
| grep -v grep | wc -l

Expected output:

0

4.2 Back Up NiFi Configurations and State

1. Create the backup directory.

mkdir -p "$B"

2. Back up the NiFi configurations and state.

tar -czvf "$B/nifi1-$TS.tar.gz" \
  /etc/nifi/conf \
  /var/lib/nifi/conf \
/var/lib/nifi/state

3. The backup contains the following files, among others:

flow.json.gz
flow.xml.gz
nifi.properties
bootstrap.conf

4. The backup archive was approximately 24 KB in the verified environment.

4.3 Back Up NiFi Registry

Run the following command:

tar -czvf "$B/registry1-$TS.tar.gz" \
  /etc/nifi-registry/conf \
  /var/lib/nifi-registry/conf \
  /var/lib/nifi-registry/database \
  /var/lib/nifi-registry/flow_storage \
  /var/lib/nifi-registry/extension_bundles

The backup includes the Registry database and six flow snapshots.

The archive was approximately 28 KB in the verified environment.

4.4 Export Ambari Service Configurations

Export the existing Ambari configurations for NiFi and NiFi Registry.

for S in NIFI NIFI_REGISTRY; do
 curl -s -u "$AMBARI_USER:$AMBARI_PASSWORD" \
 -H 'X-Requested-By: ambari' \
 "http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/configurations/service_config_versions?service_name=$S&is_current=true" \
 > "$B/ambari-$S-configs-$TS.json"
done

The exported configurations included:

Both exports included Ranger-related configuration types.

4.5 Verify Backup Integrity

1. Verify that the backup archives are readable.

for f in "$B"/*.tar.gz; do
  tar -tzf "$f" >/dev/null \
    && echo "OK $f" \
    || echo "BAD $f"
done

Expected output:

OK nifi1-...tar.gz
OK registry1-...tar.gz

2. Generate SHA-256 checksums.

cd "$B"
sha256sum * | tee SHA256SUMS.txt

5. Upgrade ODP and Remove NiFi 1

This section describes the ODP upgrade and removal of the existing NiFi services.

5.1 Upgrade ODP

1. Run the ODP Express Upgrade procedure to upgrade from ODP 3.3.6.5 to ODP 3.4.3.0.
2. In Ambari, delete the existing NIFI and NIFI_REGISTRY services.
3. Uninstall the NiFi 1 MPack.
4. Restore the environment variables if the shell session was reset during the upgrade.

5.2 Verify Backup Integrity After the Upgrade

Run the following commands:

cd "$B"
sha256sum -c SHA256SUMS.txt
ls /usr/odp/

All four checksums: OK

/usr/odp:
3.3.6.5
3.4.3.0
current

5.3 Verify the ODP Stack Version

Run the following command:

curl -s -u "$AMBARI_USER:$AMBARI_PASSWORD" \
 "http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME?fields=Clusters/version" \
 | grep version

"version" : "ODP-3.4"

5.4 Verify NiFi Service Removal

Run the following command:

curl -s -u "$AMBARI_USER:$AMBARI_PASSWORD" \
 "http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services" \
 | grep service_name \
 | grep -i nifi \
 || echo "no NiFi services registered"

no NiFi services registered

5.5 Verify the Available MPacks

After removing the NiFi 1 MPack, verify that the NiFi 2 MPack remains available.

nifi2-ambari-mpack-2.8.0.3.4.3.0-1014

The ODP 3.4 stack provides the following services:

NIFI2
NIFI_REGISTRY2

6. Install NiFi 2 and NiFi Registry 2

6.1 Add the Services

1. Log in to the Ambari UI.
2. Open the Add Service wizard.
3. Add NIFI2 and NIFI_REGISTRY2.
4. Configure the required service properties.
5. Enter the sensitive properties key used by NiFi 1.
6. Complete the installation.

The sensitive properties key was reported to be the same as the NiFi 1 key, but it was not independently verified.

After installation, both services were in the INSTALLED state, with no running processes.

The following directories had not yet been created:

/var/lib/nifi2
/var/lib/nifi-registry2

/etc/nifi2/conf

6.2 Configure the NiFi Flow File Path

An issue was identified in the NiFi 2 MPack configuration.

The MPack template configured the following property:

nifi.flow.configuration.file={{nifi_flow_config_dir}}/flow.xml.gz

Workaround:

1. In Ambari, navigate to NIFI2 > Configs > nifi2-properties.
2. Locate nifi.flow.configuration.file.
3. Change its value to:

{{nifi_flow_config_dir}}/flow.json.gz

4. Save the configuration before starting NiFi 2.

The resolved configuration should be:

nifi.flow.configuration.file=/var/lib/nifi2/conf/flow.json.gz

6.3 Verify the Configuration Directories

Run the following command to inspect the configured directories:

for S in NIFI2 NIFI_REGISTRY2; do
 curl -s -u "$AMBARI_USER:$AMBARI_PASSWORD" \
 "http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/configurations/service_config_versions?service_name=$S&is_current=true" \
 | grep -oE '"[a-z0-9_.-]*(config\.dir|database\.dir|internal\.dir|repository\.dir[a-z.]*|state\.dir|flow\.config[a-z.]*)" *: *"[^"]*"' \
 | sort -u
done

nifi.flow.configuration.file:
{{nifi_flow_config_dir}}/flow.json.gz

nifi.flow.config.dir:
{nifi_internal_dir}/conf

nifi.internal.dir:
/var/lib/nifi2

nifi.registry.internal.dir:
/var/lib/nifi-registry2

nifi.registry.database.dir:
{nifi_registry_internal_dir}/database

6.4 Identify the Data Restoration Paths

The following directories are used to restore NiFi and Registry data.

6.5 Initialize the Services

1. Start NIFI_REGISTRY2 in Ambari.
2. Start NIFI2.
3. Verify that the required directories and service users have been created.
4. Verify that NiFi Registry 2 is initially empty.

curl -s http://$(hostname -f):62080/nifi-registry-api/buckets

[]

5. Stop both services before restoring the backed-up data.
6. Verify that no NiFi processes remain.

ps aux | grep -i 'org.apache.nifi' \
| grep -v grep | wc -l

7. Expected output:

0

6.6 Verify NiFi Properties and Registry Database Compatibility

Verify the NiFi configuration properties.

grep -E '^nifi.flow.configuration.file=|^nifi.sensitive.props.algorithm=' \
  /etc/nifi2/conf/nifi.properties

Expected output:

nifi.flow.configuration.file=/var/lib/nifi2/conf/flow.json.gz
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256

The H2 libraries were packaged inside the Registry web application archives rather than directly under the lib directory.

Both database files used the following format:

MVStore format:3

/var/lib/nifi2/conf/flow.json.gz
Size: 543 bytes

/var/lib/nifi-registry2/database/*.mv.db
Size: 798720 bytes

7. Restore NiFi and NiFi Registry Data

Important: Ensure that NIFI2 and NIFI_REGISTRY2 are stopped before restoring the backed-up data.

7.1 Restore the NiFi Flow Configuration

1. Preserve the newly generated flow configuration.

mv /var/lib/nifi2/conf/flow.json.gz \ /var/lib/nifi2/conf/flow.json.gz.fresh-install

2. Restore the NiFi 1 flow configuration.

tar -xzOf "$B/nifi1-$TS.tar.gz" \
  var/lib/nifi/conf/flow.json.gz \
 > /var/lib/nifi2/conf/flow.json.gz

3. Update file ownership and permissions.

chown nifi2:hadoop \
 /var/lib/nifi2/conf/flow.json.gz

chmod 644 \
/var/lib/nifi2/conf/flow.json.gz

4. Verify the restored flow components.

zcat /var/lib/nifi2/conf/flow.json.gz \
  | grep -o '"componentType":"[A-Z_]*"' \
  | sort \
  | uniq -c

Expected output:

6 CONNECTION
1 FLOW_REGISTRY_CLIENT
6 PROCESS_GROUP
12 PROCESSOR

7.2 Extract the NiFi Registry Backup

Create a temporary restoration directory and extract the Registry database and flow snapshots.

mkdir -p /var/tmp/reg1-restore

tar -xzf "$B/registry1-$TS.tar.gz" \
  -C /var/tmp/reg1-restore \
  var/lib/nifi-registry/database \
  var/lib/nifi-registry/flow_storage

7.3 Preserve the Newly Initialized Registry Data

Rename the existing Registry 2 directories.

mv /var/lib/nifi-registry2/database \
  /var/lib/nifi-registry2/database.fresh-install

mv /var/lib/nifi-registry2/flow_storage \
  /var/lib/nifi-registry2/flow_storage.fresh-install

7.4 Restore the Registry Database and Flow Snapshots

1. Restore the Registry database.

cp -a \
  /var/tmp/reg1-restore/var/lib/nifi-registry/database \
/var/lib/nifi-registry2/database

2. Restore the flow snapshots.

cp -a \
  /var/tmp/reg1-restore/var/lib/nifi-registry/flow_storage \
/var/lib/nifi-registry2/flow_storage

3. Update ownership.

chown -R nifiregistry2:hadoop \
  /var/lib/nifi-registry2/database \
/var/lib/nifi-registry2/flow_storage

4. Verify the restored snapshots.

find /var/lib/nifi-registry2/flow_storage \
-name '*.snapshot' | wc -l

Expected output:

6

8. Start and Verify the Upgraded Services

8.1 Start NiFi Registry 2 and NiFi 2

1. In Ambari, start NIFI_REGISTRY2.
2. Start NiFi 2.
3. Verify that both services are running.

8.2 Verify Registry Buckets

Run the following command:

curl -s \
 http://$(hostname -f):62080/nifi-registry-api/buckets

Bucket: odp_smoke_bucket_1791206172
ID: bb1fc589-5342-4646-a621-a3e9dd75b897

No ERROR entries were reported in nifi-registry-app.log during the verification.

8.3 Verify Restored NiFi Components

Check the NiFi application log.

grep 'Flow contains the following number of components' \
  /var/log/nifi2/nifi-app.log | tail -1

Processors: 12
Process Groups: 5 (+ root)
Flow Registry Clients: 1

8.4 Identify Unsupported Processors

Run the following command to identify processors that NiFi 2 could not create.

grep -oE 'Could not create Processor of type [A-Za-z0-9_.]+' \
  /var/log/nifi2/nifi-app.log \
  | sort \
  | uniq -c

Seven of the twelve processors loaded successfully. ConvertJSONToSQL remained available.

Important: Restoring the flow configuration does not guarantee that all NiFi 1 processors are compatible with NiFi 2. Review the unavailable processors and replace or migrate them to supported alternatives before running the affected flows.

8.5 Update the Registry Client URL

After restoration, the Registry client continued to reference the NiFi Registry 1 endpoint on port 61080.

This caused synchronization errors for the following process groups:

testgrp1
testgrp3
kafka

To resolve the issue:

1. Open the NiFi 2 UI.
2. Navigate to Settings > Registry Clients.
3. Locate the Registry client named reg1.
4. Change the Registry URL to:

http://odp3365-0.odp3365.prabhjyot.svc.cluster.local:62080

5. Save the changes.
6. Restart NiFi 2.

8.6 Verify the Registry Client Configuration

Run the following command to retrieve the Registry client details.

curl -s \
 http://$(hostname -f):9190/nifi-api/controller/registry-clients \
 | grep -oE '"(name|url)":"[^"]*"'

Check that the Registry URL was persisted in the flow configuration.

zcat /var/lib/nifi2/conf/flow.json.gz \
  | grep -o '"url":"[^"]*"' \
  | sort -u

8.7 Verify Flow Version Synchronization

After restarting NiFi 2, the following process groups reported the expected version-control states.

The process groups without version-control states had not been versioned.

The Registry contained the following flows:

8.8 Verify Access to a Versioned Flow

Use the Registry API to verify that version 2 of the restored flow is accessible.

curl -s -o /dev/null -w '%{http_code}\n' \
 "http://$(hostname -f):62080/nifi-registry-api/buckets/bb1fc589-5342-4646-a621-a3e9dd75b897/flows/7425439e-062e-4546-ab10-1dfe57ecdb76/versions/2"

200

9. Run the Post-Upgrade Checks

After restoring and validating the services, run the post-upgrade checks from Ambari.

9.1 NiFi 2 Post-Upgrade Check

The NIFI2 POST_UPGRADE_CHECK command initially failed.

FAIL /nifi-api/access/config
HTTP 404
Expected: HTTP 200

OK /nifi-api/system-diagnostics
HTTP 200

The /nifi-api/access/config endpoint is not supported in NiFi 2.8.0.

The authentication configuration is available through:

/nifi-api/authentication/configuration

9.2 Verify NiFi 2 API Endpoints

The following endpoints were verified.

9.3 NiFi Registry 2 Post-Upgrade Check

The Registry post-upgrade check also referenced an unsupported endpoint.

/nifi-registry-api/access/config

The following Registry endpoints returned HTTP 200:

/nifi-registry-api/buckets
/nifi-registry-api/access
/nifi-registry-api/config
/nifi-registry-api/about
/nifi-registry-api/items

9.4 Resolve the Post-Upgrade Check Failures

The post-upgrade validation scripts required the following endpoint changes.

For NiFi Registry 2, the updated check accepts HTTP 200 or 401. The behavior in secured environments was not tested.

The scripts were patched locally in the Ambari agent cache, and the post-upgrade checks were reported to work.

Note: The exact steps and output from the successful checks were not captured. The changes were local and had not been committed to the MPack repository at the time of verification.

10. MPack Configuration Changes

The following changes were identified for the NiFi 2 and NiFi Registry 2 MPacks in the ambari-mpacks repository on the ODP-3.4.x-main branch.

These changes addressed the flow configuration path and post-upgrade validation issues identified during testing.

11. Post-Upgrade Cleanup

The following files and directories were retained on the host after the upgrade.

Retain the original backup directory for recovery purposes.

After validating the upgraded services and confirming that rollback is no longer required, review the temporary files and directories according to the organization's retention policy.

12. Known Issues and Limitations

The following issues were identified during the upgrade verification.

12.1 NiFi 2 Does Not Load the Restored JSON Flow

Issue: The NiFi 2 MPack configured nifi.flow.configuration.file to reference flow.xml.gz. As a result, NiFi 2 did not load the restored JSON flow configuration.

Workaround: In Ambari, navigate to NIFI2 > Configs > nifi2-properties and set the property to:
{{nifi_flow_config_dir}}/flow.json.gz

12.2 Some NiFi 1 Processors Are Not Available in NiFi 2

Issue: Some processors from the NiFi 1 flow could not be instantiated after the upgrade because their processor implementations were unavailable in NiFi 2.

The affected processor types included:

ConsumeKafka_2_6
PublishKafka_2_6
PublishKafkaRecord_2_6
ConvertAvroToJSON

12.3 Registry Synchronization Fails After Restoration

Issue: The restored NiFi flow retained the NiFi Registry 1 URL, which referenced port 61080. This caused connection failures when NiFi 2 attempted to synchronize versioned flows.

Workaround: Update the Registry client URL in NiFi 2 > Settings > Registry Clients to use the NiFi Registry 2 endpoint on port 62080. Restart NiFi 2 and verify that the versioned process groups report the UP_TO_DATE state.

12.4 NiFi 2 Post-Upgrade Check Fails with HTTP 404

Issue: The NiFi 2 post-upgrade validation script queried /nifi-api/access/config, which is not available in NiFi 2.8.0.

Resolution identified during verification: Update the validation script to query:

/nifi-api/authentication/configuration

12.5 NiFi Registry 2 Post-Upgrade Check Fails with HTTP 404

Issue: The Registry post-upgrade validation script queried /nifi-registry-api/access/config, which returned HTTP 404.

Resolution identified during verification: Update the validation script to query:

/nifi-registry-api/access

13. Upgrade Verification Summary

The following table summarizes the results of the upgrade verification.

The upgrade verification confirmed that NiFi flows, Registry metadata, and versioned flow snapshots could be migrated from ODP 3.3.6.5 to ODP 3.4.3.0 in the tested single-node environment.