Working with Management Packs

Management packs allow you to deploy a range of services to your Ambari-managed cluster. You can use a management pack to deploy a specific component or service.

Accessing ODP Repositories

The ODP mpacks can be accessed via the links listed below:

Installing Management Pack

In general, when working on installing management packs, perform the following steps :

1. Download Mpack file.
2. Install the management pack using ambari-server install-mpack --mpack=[service_mpack.gz] --verbose.

#Install Ambari mpack

ambari-server install-mpack --mpack=[service_mpack.gz] --verbose

3. Restart the Ambari Server ambari-server restart.
4. Add service from Ambari UI.
5. Select host(s) for respective component(s).
6. Start all stopped or required services.

Uninstalling Management Pack

To uninstall ambari mpack, perform the following steps:

1. Login to Ambari service UI and go to respective service section.
2. Stop respective service and delete permanently.
3. Uninstall the service from CLI using ambari-server uninstall-mpack --mpack-name=[service_mpack.gz].

#Uninstall Ambari mpack

ambari-server uninstall-mpack --mpack-name=[service_mpack.gz]

4. Restart the Ambari Server ambari-server restart.

Hue

Install Hue with Ambari using the following management pack created by Acceldata for ODP clusters.

• Version Key: hue-4.11.0.3.2.2.0-2.tar.gz - Hue-4.11.0 mpack, compatible with ODP 3.2.3.3-3 release.
• Usage Notes: Minimal installation requires Ambari, HDFS, Hive, Tez, Yarn, and Zookeeper.

Install Hue Mpack

To install Ambari Hue Mpack, perform the following:

1. Download the branch from the code and zip with tar.gz extension.
2. Upload the gzipped file to the ambari-server.
3. Execute the below command for installation:

ambari-server install-mpack --mpack=ambari-mpacks-hue-4.11.0.tar.gz --verbose

ambari-server restart

4. Login to the Ambari UI and navigate to Add Hue service.
5. Select the hosts for Hue Server components.
6. Start all stopped or required services.

Hue Custom Database

To create a Hue custom database, perform the following:

1. Get the database details.
2. Create the Hue database and configure the Hue user along with your own password.

Example Using Hue with MySQL

1. Create a new user in MySQL:

• Log in to MySQL as the root user:

mysql -u root -p

• Create a new database for Hue:

CREATE DATABASE hue;

• Create a new user for Hue:

CREATE USER $HUEUSER IDENTIFIED BY '$HUEPASSWORD';

• Grant all privileges to the new user on all databases:

GRANT ALL PRIVILEGES ON *.* TO '$HUEUSER'@'localhost' WITH GRANT OPTION;

GRANT ALL ON $HUEUSER.* TO '$HUEUSER'@'localhost' IDENTIFIED BY '$HUEPASSWORD';

GRANT ALL ON $HUEUSER.* TO '$HUEUSER'@'%' IDENTIFIED BY '$HUEPASSWORD';

• Flush privileges to apply changes:

FLUSH PRIVILEGES;

Note Ensure to replace $HUEUSER with the desired Hue username and $HUEPASSWORD with the desired Hue user password.

Configure Hue Mpack

The Impala and HDFS services can be automatically configured by the Hue mpack. To enable auto-configuration for these services, ensure their respective service modules are enabled in the Ambari UI.

If the modules are not enabled, the auto-configuration does not occur. You may still manually configure the services, if preferred.

If additional configurations are needed for any of these services, their configuration templates are exposed in the Ambari UI under ‘Advanced services’. These configuration files can be found in /etc/hue/conf as hdfs.ini and impala.ini if their respective service is enabled.

Uninstall Hue Mpack

To uninstall Ambari Hue Mpack, perform the following steps:

1. Login to Ambari Service and Navigate to Hue Section:

   • Access the Ambari service web interface.
   • Navigate to the Hue section in the Ambari dashboard.

2. Stop the Hue Service and Delete Permanently:

   • Locate the option to stop the Hue service in the Ambari dashboard.
   • Stop the Hue service and confirm the action.
   • Delete the Hue service permanently from the Ambari dashboard.

3. Open a terminal or command prompt on the Ambari server.

   • Run the following command to uninstall the Hue Mpack:

ambari-server uninstall-mpack --mpack-name=hue-ambari.mpack

Replace hue-ambari.mpack with the actual name of the Hue Mpack if it differs.

4. Restart the Ambari Server:
   • After uninstalling the Hue Mpack, restart the Ambari server to apply the changes.

ambari-server restart

These steps will uninstall the Ambari Hue Mpack from your Ambari service.

Troubleshooting Steps for Hue Known Issues

Issue 1: If you see the following error message in the Hue logs, set the configuration mentioned in the solution proposed below.

File "/usr/lib/python3.11/tempfile.py", line 396, in _mkstemp_inner

 fd = _os.open(file, flags, 0o600)

      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

PermissionError: [Errno 13] Permission denied: '/usr/lib/hue/desktop/conf/wgunicorn-9zlljl_m'

Solution: Uncomment and set the following config as shown in Advanced hue.ini.

# Webserver runs as this user

  server_user=hue

  server_group=hadoop

Issue 2: Unable to find the mysql-devel package for Hue in RHEL 9.

File "/usr/lib/ambari-agent/lib/ambaricommons/shell.py", line 750, in repository manager_executor raise RuntimeError(message) RuntimeError: Failed to execute command '/usr/bin/yum -y install mysql-devel', exited with code '1', message: 'Error: Unable to find a match: mysql-devel '

Solution: Enable the CRB repository on the node where Hue is installed to fulfill the mysql-devel requirement, as it is provided by this repository.

sudo dnf config-manager --set-enabled crb

Troubleshooting Hue KT Renewer Kerberos Issue

If you come across the following error message in Hue, there are chances that the max_renewable_life and max_life properties will be missing. Make sure to add these properties to the configuration file.

[17/Mar/2025 10:46:54 +0000] decorators   INFO     AXES: blocking by IP only.

[17/Mar/2025 10:46:55 +0000] __init__     INFO     Couldn't import snappy. Support for snappy compression disabled.

[17/Mar/2025 10:46:55 +0000] kt_renewer   INFO     Reinitting kerberos retry attempt 0 from keytab /usr/bin/kinit -k -t /etc/security/keytabs/hue.headless.keytab -c /tmp/krb5cc_1028 hue-ub22-32332@ADSRE.COM

[17/Mar/2025 10:46:56 +0000] kt_renewer   INFO     Renewing kerberos ticket to work around kerberos 1.8.1: /usr/bin/kinit -R -c /tmp/krb5cc_1028

kinit: KDC can't fulfill requested option while renewing credentials

[17/Mar/2025 10:46:56 +0000] kt_renewer   ERROR    Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue-ub22-32332@ADSRE.COM' is still renewable:

  $ klist -f -c /tmp/krb5cc_1028

If the 'renew until' date is the same as the 'valid starting' date, the ticket cannot be renewed. Please check your KDC configuration, and the ticket renewal policy (maxrenewlife) for the 'hue-ub22-32332@ADSRE.COM' and `krbtgt' principals.

For CentOS/RHEL:

1. On the KDC host, add the max_renewable_life and max_life properties in the configuration file.

[root@ce7-f1-node01 ~]# cat /var/kerberos/krb5kdc/kdc.conf

[realms]

    ADSRE.COM = {

        database_name = /var/kerberos/krb5kdc/principal

        acl_file = /var/kerberos/krb5kdc/kadm5.acl

        supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal des3-cbc-sha1:normal des-cbc-md5:normal arcfour-hmac-md5:normal

        max_renewable_life = 7d 0h 0m 0s           ## add    

        max_life = 12h 0m 0s                       ## add

    }

2. Once added and saved, restart KDC and KAdmin (use the following commands) and regenerate keytabs followed by restarting the Hue service from the Ambari UI.

sudo systemctl restart krb5kdc

sudo systemctl restart kadmin

For Ubuntu:

1. On the KDC host, add the max_renewable_life and max_life properties in the configuration file.

root@ub22-f1-node01:~# cat /etc/krb5kdc/kdc.conf

[realms]

    ADSRE.COM = {

        database_name = /var/lib/krb5kdc/principal

        acl_file = /etc/krb5kdc/kadm5.acl

        max_renewable_life = 7d 0h 0m 0s        ## add

        max_life = 12h 0m 0s                    ## add

    }

2. Once added and saved, restart KDC and KAdmin (use the following commands) and regenerate keytabs followed by restarting the Hue service from the Ambari UI.

sudo systemctl restart krb5-kdc

sudo systemctl restart krb5-admin-server

In addition to the KDC configuration changes, make sure to run these commands on the Kerberos Key Distribution Center (KDC) using the kadmin.local or kadmincommand-line interface:

# Allow renewable tickets for hue principal

modprinc -maxrenewlife "90 days" +allow_renewable <principal-name>@<REALM>

​

# Set renewal policy for the realm's Ticket Granting Ticket (TGT)

modprinc -maxrenewlife "90 days" krbtgt/<REALM>

​

eg

modprinc -maxrenewlife "90 days" +allow_renewable hue@ADSRE.COM

modprinc -maxrenewlife "90 days" krbtgt/ADSRE.COM

Impala

Install Impala with Ambari using the following management pack created by Acceldata for ODP clusters.

Install Impala Mpack

To install Impala Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-impala-4.4.0.tar.gz -v

4. Restart the Ambari server:

ambari-server restart

5. Log into the Ambari UI and navigate to Add Impala service.
6. Select hosts for the Catalog and Statestore components.
7. Choose daemons from the host list and proceed with service deployment to complete the installation.
8. Start any stopped or required services to finalize the installation.

Uninstall Impala Mpack

To uninstall Ambari Impala Mpack, perform the following:

1. Log in to Ambari Service and Navigate to Impala Section:

   • Access the Ambari service web interface.
   • Navigate to the Impala section in the Ambari dashboard.

2. Stop the Impala Service and Delete Permanently:

   • Locate the option to stop the Impala service in the Ambari dashboard.
   • Stop the Impala service and confirm the action.
   • Delete the Impala service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Impala Mpack:

ambari-server uninstall-mpack --mpack=ambari-impala-mpack -v

Replace ambari-impala-mpack with the actual name of the Impala Mpack if it differs.

4. Restart the Ambari Server:
   • After uninstalling the Impala Mpack, restart the Ambari server to apply the changes:

ambari-server restart

These steps will uninstall the Ambari Impala Mpack from your Ambari service.

Impala Authorization

Impala supports the Ranger-based authorization. In the Ranger Service Manager, users can leverage the preloaded Hadoop SQL resource-based and tag-based services and policies to manage Impala access control. To ensure automatic population of service caches and policies, install the Hive Client on nodes hosting the Impala components. Additionally, configure and validate the Ranger Hadoop SQL service to enable the Impala user to download policies.

Troubleshooting Steps for Impala Known Issues

1. Issue 1: Ranger Authorization Enablement

The file ranger-hive-security.xml is missing in all daemons except on Hive servers, leading to start failures. Additionally, if audit is enabled, the file ranger-hive-audit.xml is also unavailable.

F0214 14:55:06.908636 40022 frontend.cc:148] InternalException: Unable to instantiate authorization provider: org.apache.impala.authorization.ranger.RangerAuthorizationFactory

CAUSED BY: InvocationTargetException: null

CAUSED BY: IllegalArgumentException: bound must be positive

. Impalad exiting.

Solution

Copy the ranger-hive-security.xml and ranger-hive-audit.xml files from hive-servers to /etc/hive/conf/ as enabled with Ranger.

2. Issue 2: Solr Audit Enablement

When Solr is enabled for Ranger audit, daemons fail to start on infra-solr-clients due to broken symlinks in the underlying impala-solr libraries.

Solution

To resolve the issue of broken symlinks for Solr in the impala-solr libraries, install ambari-infra-solr on all daemons.

3. Issue 3: Missing Ranger Cache Policies

Missing cache policies on other launched daemons or Impala components.

2023-02-06T10:19:21,074 ERROR [main] org.apache.ranger.plugin.contextenricher.RangerTagEnricher$RangerTagRefresher - failed to save service-tags to cache file '/etc/ranger/adodp_hive/policycache/impala_adodp_hive_tag.json'

java.io.FileNotFoundException: /etc/ranger/adodp_hive/policycache/impala_adodp_hive_tag.json (No such file or directory)

at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_342]

at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_342]

Solution

Create Ranger policy cache directories under /etc/ranger/conf and copy the policy cache files from the respective policy directories.

Accessing Impala Web UI Consoles

Each of the Impala daemons (impalad, statestored, and catalogd) includes a built-in web server that displays diagnostic and status information.

Catalogd WebUI Console

The catalogd Web UI includes information about the databases, tables, and other objects managed by Impala, in addition to the resource usage and configuration settings of the catalogd. Because there is only a single instance of the catalogd within any Impala cluster, you access the Web UI only on the particular host that serves as the Impala Catalog Server.

http://<catalog-host-name>:25020

Statesoted WebUI Console

The statestored Web UI includes information about memory usage, configuration settings, and ongoing health checks performed by statestored. Because there is only a single instance of the statestored within any Impala cluster, you can access the Web UI only on a particular host that serves as the Impala StateStore.

http://<Statestore-host-name>:25010

Impald WebUI Console

The impalad Web UI includes information about configuration settings, running and completed queries, and associated performance and resource usage for queries. In particular, the Details link for each query displays alternative views of the query including a graphical representation of the plan, and the output of the EXPLAIN, SUMMARY, and PROFILE statements from impala-shell. Each host that runs the impalad daemon has its own instance of the Web UI, with details about those queries for which that host served as the coordinator. The impalad Web UI is primarily used for diagnosing query problems that can be traced to a particular node.

http://<Impalad-host-name>:25000

For more information about accessing the Impala Web interface, see Impala Web User Interface for Debugging.

NiFi

Install NiFi with Ambari using the following management pack created by Acceldata for ODP clusters.

Version Key

• nifi-1.23.2.tar.gz - Nifi-1.23.2 mpack

Usage Notes

• Minimal installation requires Ambari, HDFS, and Zookeeper.

Install NiFi Mpack

To install Ambari NiFi Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-nifi-1.23.2.tar.gz --verbose

4. Restart the Ambari server:

ambari-server restart

5. Log in to the Ambari UI and navigate to Add NiFi service.
6. Select hosts for the NiFi components.
7. Start any stopped or required services to finalize the installation.

Uninstall NiFi Mpack

To uninstall Ambari NiFi Mpack, perform the following steps:

1. Login to Ambari Service and Navigate to NiFi Section:

   • Access the Ambari service web interface.
   • Navigate to the NiFi section in the Ambari dashboard.

2. Stop the NiFi Service and Delete Permanently:

   • Locate the option to stop the NiFi service in the Ambari dashboard.
   • Stop the NiFi service and confirm the action.
   • Delete the NiFi service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Impala Mpack:

ambari-server uninstall-mpack --mpack-name=nifi-ambari-mpack

4. Restart the Ambari Server:
   • After uninstalling the NiFi Mpack, restart the Ambari server to apply the changes:

ambari-server restart

These steps will uninstall the Ambari NiFi Mpack from your Ambari service.

Troubleshooting Steps for NiFi Known Issues

1. Issue 1: User Group Management for Ambari 2.8 & ODP 3.x.

Solution

Run the following Python code before installing NiFi:

/var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n [CLUSTER_NAME] -l [CLUSTER_FQDN] -t 8080 -a set -c cluster-env -k  ignore_groupsusers_create -v true

Replace the CLUSTER_NAME and CLUSTER_FQDN with the correct names.

Kafka 3

Install Kafka 3 with Ambari using this management pack created by Acceldata for ODP clusters.

Install Kafka 3 Mpack

To install Ambari Kafka 3 Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack.

ambari-server install-mpack --mpack=ambari-mpacks-kafka3-3.7.1.tar.gz --verbose

After running above commands you can see this message.

INFO: Management pack kafka3-ambari-mpack-3.7.1 successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

The Ambari server restarts.

#Restart Ambari server

ambari-server restart

4. Log in to the Ambari UI and navigate to Add Kafka3 service.
5. Select hosts for the Kafka 3 History Server and client components.
6. Start any stopped or required services to finalise the installation.

Uninstall Kafka 3 Mpack

To uninstall the Ambari Kafka 3 Mpack, perform the following steps:

1. Log in to the Ambari service and navigate to the Kafka 3 section:

   • Access the Ambari service web interface.
   • Navigate to the Kafka 3 section in the Ambari dashboard.

2. Stop the Kafka 3 service and delete it permanently:

   • Locate the option to stop the Kafka 3 service in the Ambari dashboard.
   • Stop the Kafka 3 service and confirm the action.
   • Delete the Kafka 3 service permanently from the Ambari dashboard.

3. Execute the Uninstallation command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Kafka 3 Mpack.

ambari-server uninstall-mpack --mpack-name=kafka3-ambari-mpack

4. Restart the Ambari server:
   • After uninstalling the Kafka 3 Mpack, restart the Ambari server to apply the changes.

These steps uninstall the Ambari Kafka 3 Mpack from your Ambari service.

Flink

Install Flink with Ambari using this management pack created by Acceldata for ODP clusters.

Install Flink Mpack

To install Ambari Flink Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack.

ambari-server install-mpack --mpack=ambari-mpacks-flink-1.19.1.tar.gz --verbose

After running above commands you can see this message.

INFO: Management pack flink-ambari-mpack-1.19.1 successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

The Ambari server restarts.

#Restart Ambari server

ambari-server restart

4. Log into the Ambari UI and navigate to Add Flink service.
5. Select hosts for the Flink History Server and client components.
6. Start any stopped or required services to finalise the installation.

Uninstall Flink Mpack

To uninstall the Ambari Flink Mpack, perform the following steps:

1. Log in to the Ambari service and navigate to the Flink section:

   • Access the Ambari service web interface.
   • Navigate to the Flink section in the Ambari dashboard.

2. Stop the Flink service and delete it permanently:

   • Locate the option to stop the Flink service in the Ambari dashboard.
   • Stop the Flink service and confirm the action.
   • Delete the Flink service permanently from the Ambari dashboard.

3. Execute the Uninstallation command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Flink Mpack.

ambari-server uninstall-mpack --mpack-name=flink-ambari-mpack

4. Restart the Ambari server:
   • After uninstalling the Flink Mpack, restart the Ambari server to apply the changes.

ambari-server restart

These steps uninstall the Ambari Flink Mpack from your Ambari service.

Spark 3

Install Spark3 with Ambari using this management pack created by Acceldata for ODP clusters.

Version Key

• spark3_mpack-3.5.1.tar.gz - Spark3.5.1 mpack, compatible with ODP 3.2.3.3-3 release.

Usage Notes

• Minimal installation requires Ambari, HDFS, Hive, Tez, Yarn, and Zookeeper.

Install Spark 3 Mpack

To install Ambari Spark 3 Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-spark3-3.5.1.tar.gz --verbose

In case you find the following errors, run the following commands as mentioned:

(mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode)

  File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 798, in _install_mpack

    process_stack_addon_service_definitions_artifact(artifact, artifact_source_dir, options)

  File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 557, in process_stack_addon_service_definitions_artifact

    sudo.symlink(source_service_version_path, dest_link)

  File "/usr/lib/ambari-server/lib/resource_management/core/sudo.py", line 130, in symlink

    os.symlink(source, link_name)

OSError: [Errno 17] File exists

​

ambari-server uninstall-mpack --mpack-name=spark3-ambari-3.5.1.mpack

​

rm -rf /var/lib/ambari-server/data/tmp/ambari-mpacks-spark3-3.5.1*/ /var/lib/ambari-server/resources/stacks/ODP/3.0/services/SPARK3 /var/lib/ambari-server/resources/stacks/ODP/3.1/services/SPARK3 /var/lib/ambari-server/resources/common-services/SPARK3 /var/lib/ambari-agent/cache/stacks/ODP/3.0/services/SPARK3 /var/lib/ambari-agent/cache/stacks/ODP/3.1/services/SPARK3 /var/lib/ambari-agent/cache/common-services/SPARK3 /var/lib/ambari-agent/cache/stacks/ODP/3.2/services/SPARK3 /var/lib/ambari-server/resources/stacks/ODP/3.2/services/SPARK3

Incase of the above failures, run the installation command:

ambari-server install-mpack --mpack=ambari-mpacks-spark3-3.5.1.tar.gz --verbose

​

cd /var/lib/ambari-server/resources/stacks/ODP/3.0/services/

rm -f SPARK3

ln -s /var/lib/ambari-server/resources/mpacks/spark3-ambari-3.5.1.mpack-3.5.1/common-services/SPARK3/3.5.1 SPARK3

cd ../../3.1/services/

rm -f SPARK3

ln -s /var/lib/ambari-server/resources/mpacks/spark3-ambari-3.5.1.mpack-3.5.1/common-services/SPARK3/3.5.1 SPARK3

​

ambari-server restart

After running the above commands, the following message is shown:

INFO: Management pack spark3-ambari-3.5.1.mpack-3.5.1 successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

4. Log in to the Ambari UI and navigate to Add Spark 3 service.
5. Select hosts for the Spark thrift and client components.
6. Start any stopped or required services to finalize the installation.

Uninstall Spark 3 Mpack

To uninstall Ambari Spark 3 Mpack, perform the following steps:

1. Login to Ambari Service and Navigate to Spark 3 Section:

   • Access the Ambari service web interface.
   • Navigate to the Spark 3 section in the Ambari dashboard.

2. Stop the Spark 3 Service and Delete Permanently:

   • Locate the option to stop the Spark 3 service in the Ambari dashboard.
   • Stop the Spark 3 service and confirm the action.
   • Delete the Spark 3 service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Spark 3 Mpack:

ambari-server uninstall-mpack --mpack-name=spark3-ambari-3.5.1.mpack

4. Restart the Ambari Server:
   • After uninstalling the Spark 3 Mpack, restart the Ambari server to apply the changes:

ambari-server restart

These steps will uninstall the Ambari Spark 3 Mpack from your Ambari service.

Troubleshooting Steps for Spark 3 Known Issues

1. Issue 1: Issue in User Group Management for Ambari 2.8 & ODP 3.x.

Solution

Run the following Python command before installing Spark 3:

/var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n [CLUSTER_NAME] -l [CLUSTER_FQDN] -t 8080 -a set -c cluster-env -k  ignore_groupsusers_create -v true

Replace the [CLUSTER_NAME] __and __[CLUSTER_FQDN] in the command above with the correct names.

HttpFS

Ambari HttpFS Service

This is a straightforward Ambari extension for the HttpFS HDFS service.

Supported Operating Systems: Any

Tested with Ambari Server 2.7.8.0.

Install HttpFS Mpack

To install Ambari HttpsFS Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-httpfs-2.7.1.tar.gz --verbose

4. Restart the Ambari Service:

ambari-server restart

5. Log in to the Ambari UI and navigate to Add HttpFS service.
6. Select hosts for the HttpFs Server components.
7. Start any stopped or required services to finalize the installation.

Uninstall HttpFS Mpack

To uninstall Ambari HttpFS Mpack, perform the following steps:

1. Login to Ambari Service and Navigate to HttpFS Section:

   • Access the Ambari service web interface.
   • Navigate to the HttpFS section in the Ambari dashboard.

2. Stop the HttpFS Service and Delete Permanently:

   • Locate the option to stop the HttpFS service in the Ambari dashboard.
   • Stop the HttpFS service and confirm the action.
   • Delete the HttpFS service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the HttpFS Mpack:

ambari-server uninstall-mpack --mpack-name=httpfs-ambari-mpack

4. Restart the Ambari Server:
   • After uninstalling the HttpFS Mpack, restart the Ambari server to apply the changes:

ambari-server restart

These steps will uninstall the Ambari HttpFS Mpack from your Ambari service.

Troubleshooting Steps for HttpFS Known Issues

Issue 1 : Bug in User Group Management for Ambari 2.8 & ODP 3.x.

Solution

Run the following Python command before installing HttpFS:

/var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n [CLUSTER_NAME] -l [CLUSTER_FQDN] -t 8080 -a set -c cluster-env -k  ignore_groupsusers_create -v true

Ensure the correct [CLUSTER_NAME] and [CLUSTER_FQDN] names are used.

License: Apache License version 2.0

Ozone

Install Ozone with Ambari using this management pack created by Acceldata. Note Ozone provided in this distribution supports HA only.

Version Key

• ozone-1.4.0.tar.gz - Ozone 1.4.0 mpack, compatible from ODP v3.2.3.0-2 release onwards

Usage Notes

• Minimal install required Ambari
• For Authentication integration it is compatible with Kerberos
• For Authorization integration it requires Ranger with auditing disabled due to performance issues in upstream.

Install Ozone Mpack

To install Ambari Ozone Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-ozone-1.4.0.tar.gz --verbose

After running the above commands, the following message is shown:

INFO: Management pack ozone-ambari-mpack-1.4.0 successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

4. Create Service Symlink directory:

cd /var/lib/ambari-server/resources/stacks/ODP/3.0/services/

rm -f OZONE

ln -s /var/lib/ambari-server/resources/mpacks/ozone-ambari.mpack-1.4.0/common-services/OZONE/1.4.0 OZONE

cd ../../3.1/services/

rm -f OZONE

ln -s /var/lib/ambari-server/resources/mpacks/ozone-ambari.mpack-1.4.0/common-services/OZONE/1.4.0 OZONE

cd ../../3.2/services/

rm -f OZONE

ln -s /var/lib/ambari-server/resources/mpacks/ozone-ambari.mpack-1.4.0/common-services/OZONE/1.4.0 OZONE

5. Restart the Ambari service:

ambari-server restart

6. Log in to the Ambari UI and navigate to add Ozone service.
7. Select at least three hosts for the Ozone Manager, three hosts for Storage Container Manager, hosts for S3 gateway, host for Recon server, and at least three nodes for Data node.
8. Configure Ozone as per your use case and deploy the service.
9. Start any stopped or required services to finalize the installation.

Uninstall Ozone Mpack

To uninstall Ambari Ozone Mpack, perform the following steps:

1. Log in to Ambari Service and Navigate to Ozone Section:

   • Access the Ambari service web interface.
   • Navigate to the Ozone section in the Ambari dashboard.

2. Stop the Ozone Service and Delete Permanently:

   • Locate the option to stop the Ozone service in the Ambari dashboard.
   • Stop the Ozone service and confirm the action.
   • Delete the Ozone service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Ozone Mpack:

ambari-server uninstall-mpack --mpack-name=ozone-ambari.mpack

4. Restart the Ambari Server:
   • After uninstalling the Ozone Mpack, restart the Ambari server to apply the changes:

ambari-server restart

Registry

Install Schema Registry with Ambari using this management pack created by Acceldata for ODP Clusters.

Version Key

• registry-1.0.0.tar.gz - Registry 1.0.0 mpack, compatible from ODP v3.2.3.1-2 release onwards.

Usage Notes

• Minimal installation requires Ambari.
• For authentication integration it is compatible with Knox, OAUTH2 and Kerberos.
• For authorization integration it requires Ranger and for audit enablement it requires HDFS.

Install Registry Mpack

To install Ambari Registry Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-registry-1.0.0.tar.gz --verbose

After running the above commands, the following message is shown:

INFO: Management pack registry-ambari-mpack-1.0.0 successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

But it requires a few symlink changes before Ambari server restart:

cd /var/lib/ambari-server/resources/stacks/ODP/3.0/services/

rm -f REGISTRY

ln -s /var/lib/ambari-server/resources/mpacks/registry-ambari-mpack-1.0.0/common-services/REGISTRY/1.0.0 REGISTRY

cd ../../3.1/services/

rm -f REGISTRY

ln -s /var/lib/ambari-server/resources/mpacks/registry-ambari-mpack-1.0.0/common-services/REGISTRY/1.0.0 REGISTRY

cd ../../3.2/services/

rm -f REGISTRY

ln -s /var/lib/ambari-server/resources/mpacks/registry-ambari-mpack-1.0.0/common-services/REGISTRY/1.0.0 REGISTRY

​

ambari-server restart

4. Log in to the Ambari UI and navigate to Add Registry service.
5. Select hosts for the Registry Server components.
6. Start any stopped or required services to finalize the installation.

Uninstall Registry Mpack

To uninstall Ambari Schema Registry Mpack, perform the following steps:

1. Login to Ambari Service and Navigate to Schema Registry Section:

   • Access the Ambari service web interface.
   • Navigate to the Registry section in the Ambari dashboard.

2. Stop the Registry Service and Delete Permanently:

   • Locate the option to stop the Registry service in the Ambari dashboard.
   • Stop the Registry service and confirm the action.
   • Delete the Registry service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Registry Mpack:

ambari-server uninstall-mpack --mpack-name=registry-ambari-mpack

4. Restart the Ambari Server:
   • After uninstalling the Registry Mpack, restart the Ambari server to apply the changes:

ambari-server restart

These steps will uninstall the Registry Mpack from your Ambari service.

Airflow

Install Apache Airflow with Ambari using this management pack created by Acceldata for ODP clusters.

Version Key

• Apache Airflow 2.8.1 - Apache Airflow 1.0.0 mpack, compatible from ODP v3.3.0 release onwards.

Usage Notes

• Minimal installation requires Ambari.
• For Authentication integration it is compatible with LDAP and Kerberos

Install Apache Airflow Mpack

To install Airflow Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-airflow-2.8.1.tar.gz --verbose

After running the above command, the following message is shown:

INFO: Management pack airflow-ambari-mpack-2.8.1 successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

Change Simlinks before restarting the Ambari Server:

cd /var/lib/ambari-server/resources/stacks/ODP/3.0/services

unlink AIRFLOW

ln -s /var/lib/ambari-server/resources/mpacks/airflow-ambari-mpack-2.8.1/common-services/AIRFLOW/2.8.1 AIRFLOW

cd /var/lib/ambari-server/resources/stacks/ODP/3.1/services

unlink AIRFLOW

ln -s /var/lib/ambari-server/resources/mpacks/airflow-ambari-mpack-2.8.1/common-services/AIRFLOW/2.8.1 AIRFLOW

cd /var/lib/ambari-server/resources/stacks/ODP/3.2/services

unlink AIRFLOW

ln -s /var/lib/ambari-server/resources/mpacks/airflow-ambari-mpack-2.8.1/common-services/AIRFLOW/2.8.1 AIRFLOW

​

ambari-server restart

4. Log in to the Ambari UI and navigate to Add Apache Airflow service.
5. Select hosts for the Apache Airflow server components.
6. Start any stopped or required services to finalize the installation.

For detailed steps about installing Airflow, see Airflow.

Uninstall Ambari Airflow Mpack

To uninstall Ambari Airflow Mpack, perform the following steps:

1. Login to Ambari Service and Navigate to Apache Airflow Section:

   • Access the Ambari service web interface.
   • Navigate to the Apache Airflow section in the Ambari dashboard.

2. Stop the Apache Airflow Servers and Delete Permanently:

   • Locate the option to stop the Apache Airflow service in the Ambari dashboard.
   • Stop the Airflow service and confirm the action.
   • Delete the Airflow service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the Apache Airflow Mpack:

ambari-server uninstall-mpack --mpack-name=airflow-ambari-mpack

4. Restart the Ambari Server:
   • After uninstalling the Airflow Mpack, restart the Ambari server to apply the changes:

ambari-server restart

JupyterHub

Install the JupyterHub with Ambari using this management pack created by Acceldata for ODP clusters.

Version Key

• Jupyterhub 5.2.1 - Jupyterhub 5.2.1 mpack, compatible from ODP v3.2.3.3-3 release onwards.

Usage Notes

• Minimal installation requires Ambari.
• For Authentication integration it is compatible with LDAP and Kerberos

Install the JupyterHub Mpack

To install the JupyterHub Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-jupyterhub-5.2.1.3.2.3.3-3.tar.gz --verbose

After running the above command, the following message is shown:

INFO: Management pack jupyter-ambari-mpack-5.2.1 successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

4. Log into the Ambari UI and navigate to add the JupyterHub service.
5. Select hosts for the JupyterHub server components.
6. Start any stopped or required services to finalize the installation.

For detailed steps about installing JupyterHub, see Jupyterhub.

Uninstall Ambari JupyterHub Mpack

To uninstall JupyterHub Mpack, perform the following steps:

1. Login to Ambari Service and Navigate to JupyterHub Section:

   • Access the Ambari service web interface.
   • Navigate to the JupyterHub section in the Ambari dashboard.

2. Stop the Apache JupyterHub Servers and Delete Permanently:

   • Locate the option to stop the JupyterHub service in the Ambari dashboard.
   • Stop the JupyterHub service and confirm the action.
   • Delete the JupyterHub service permanently from the Ambari dashboard.

3. Execute Uninstallation Command:

   • Open a terminal or command prompt on the Ambari server.
   • Run the following command to uninstall the JupyterHub Mpack:

ambari-server uninstall-mpack --mpack-name=jupyter-ambari-mpack

4. Restart the Ambari Server:
   • After uninstalling the JupyterHub Mpack, restart the Ambari server to apply the changes:

ambari-server restart

Isilon OneFS

README for Isilon Ambari Management Pack 1.0.3.0 for OneFS

The Isilon Ambari Management Pack for OneFS is a software component that integrates OneFS as a service into a Hadoop cluster via Ambari. This management pack allows an Ambari administrator to manage OneFS as a HDFS storage service, providing native namenode and datanode capabilities similar to traditional HDFS. The management pack is OneFS release-independent and can be updated between releases if necessary.

Prerequisites

This management pack is compatible with the following OneFS versions:

• OneFS 8.1.2
• OneFS 8.2.0
• OneFS 8.2.1
• OneFS 8.2.2
• OneFS 9.0.0

Install OneFS Mpack

To install the Isilon management pack, perform the following steps:

1. Run the installation command:

ambari-server install-mpack --mpack=ambari-mpacks-isilon-onefs-1.0.3.0.tar.gz --verbose

2. Update the symbolic links for the OneFS service:

cd /var/lib/ambari-server/resources/stacks/ODP/3.0/services/

rm -f ONEFS

ln -s /var/lib/ambari-server/resources/mpacks/onefs-ambari-mpack-1.0.3.0/common-services/ONEFS/1.0.0/ ONEFS

​

cd ../../3.1/services/

rm -f ONEFS

ln -s /var/lib/ambari-server/resources/mpacks/onefs-ambari-mpack-1.0.3.0/common-services/ONEFS/1.0.0/ ONEFS

​

cd ../../3.2/services/

rm -f ONEFS

ln -s /var/lib/ambari-server/resources/mpacks/onefs-ambari-mpack-1.0.3.0/common-services/ONEFS/1.0.0/ ONEFS

3. Restart the Ambari Server:

ambari-server restart

After running the above commands, you will see the following message:

INFO: Management pack onefs-ambari-mpack successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

To uninstall the management pack, run the following command:

ambari-server uninstall-mpack --mpack-name=onefs-ambari-mpack

Then, remove the following directories:

rm -rf /var/lib/ambari-server/data/tmp/isilon-onefs-mpack*/ /var/lib/ambari-server/resources/stacks/ODP/3.0/services/ONEFS /var/lib/ambari-server/resources/stacks/ODP/3.1/services/ONEFS /var/lib/ambari-server/resources/common-services/ONEFS /var/lib/ambari-agent/cache/stacks/ODP/3.0/services/ONEFS /var/lib/ambari-agent/cache/stacks/ODP/3.1/services/ONEFS /var/lib/ambari-agent/cache/common-services/ONEFS /var/lib/ambari-agent/cache/stacks/ODP/3.2/services/ONEFS

Important Notes

• A performance issue with OneFS and Ambari has been identified when Apache HBase is configured. Setting the hbase.wal.provider value to filesystem resolves this issue.
• When installing the 1.0.3.0 Ambari Management Pack on a Hadoop cluster with a new deployment of ODP, the hbase.wal.provider value will be automatically set to filesystem.
• If the Hadoop cluster already has Ambari and HBase configured, the hbase.wal.provider value must be manually configured in the Ambari UI. Follow these steps to configure it:
  1. Log in to the Ambari UI.
  2. Click the 'HBase' service in the left panel, then click the Configs tab, and finally click the Advanced tab.
  3. In the Custom hbase-site section, look for the hbase.wal.provider key. If it does not exist, add it by clicking Add Property.
  4. Enter the following in the Properties field: hbase.wal.provider=filesystem
  5. Click ADD and then Save to save the configuration. Restart Ambari if prompted to apply the setting.

Resolved Issues

• Setting the hbase.wal.provider value to filesystem in the Ambari UI resolves the performance issue.
• The OneFS Release Notes can be found on the customer support site: Support for PowerScale OneFS | Overview | Dell US
• The following documents, available on the customer support site, provide instructions for downloading and installing the Ambari Management Pack for Isilon OneFS:
  • PowerScale OneFS With Hadoop and Installation Guide
  • PowerScale OneFS With Hadoop and Hadoop for Kerberos Installation Guide

Upgrade Isilon Mpack

Run the following commands to upgrade to Isilon Ambari Management Pack 1.0.3.0 and then to restart the Ambari server:

ambari-server upgrade-mpack --mpack=

​

ambari-server restart

Kudu

Install Kudu with Ambari using this management pack created by Acceldata for ODP clusters.

Install Kudu Mpack

To install Kudu Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-kudu-1.17.0.tar.gz --verbose

4. Log on to the Ambari UI and add the Kudu service.
5. Select the hosts for Kudu Masters and Tablet Servers. You must select at least one of each, with up to a recommendation of up to 100 tablet servers. You must select an odd number for a minimum of 3 master servers.

Unistall Kudu Mpack

To uninstall Ambari Kudu Mpack, perform the following steps:

1. Log in to the Ambari UI and navigate to the Kudu section.
2. Stop the Kudu service, then delete the service.
3. Run the following command to remove the Kudu mpack.

ambari-server uninstall-mpack --mpack=kudu-ambari-mpack

4. Restart the Ambari Server to apply the changes.

ambari-server restart

Pinot

Install Pinot with Ambari using this management pack created by Acceldata for ODP clusters.

Install Pinot Mpack

To install Pinot Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-pinot-1.3.0.tar.gz --verbose

4. Log into the Ambari UI and add the Pinot service.
5. Select the hosts for Pinot Masters, Servers, Brokers, and Minions.

Uninstall Pinot Mpack

To uninstall Ambari Pinot Mpack, perform the following steps:

1. Log into the Ambari UI and navigate to the Pinot section.
2. Stop the Pinot service, then delete the service.
3. Run the following command to remove the Pinot mpack.

ambari-server uninstall-mpack --mpack=pinot-ambari-mpack

4. Restart the Ambari Server to apply the changes.

ambari-server restart

Trino

Install Trino with Ambari using this management pack created by Acceldata for ODP clusters.

Install Trino Mpack

To install Trino Mpack, perform the following steps:

1. Download the branch from the code repository and compress it into a tar.gz file.
2. Upload the compressed file to the Ambari server.
3. Execute the following command to install the Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-trino-472.tar.gz --verbose

4. Log into the Ambari UI and add the Trino service.
5. Select the hosts for Trino Masters, Servers, Brokers, and Minions.

Uninstall Trino Mpack

To uninstall Ambari Trino Mpack, perform the following steps:

1. Log into the Ambari UI and navigate to the Trino section.
2. Stop the Trino service, then delete the service.
3. Run the following command to remove the Trino mpack.

ambari-server uninstall-mpack --mpack=trino-ambari-mpack

4. Restart the Ambari Server to apply the changes.

ambari-server restart