Install Multiple Spark 3 Versions

Prerequisites

Before installing multiple Spark3 versions, ensure:

• ✅ Ambari Server 2.7.0 or higher
• ✅ ODP 3.0+ stack installed
• ✅ Required dependencies: HDFS, YARN, HIVE
• ✅ Sufficient cluster resources for multiple History Servers
• ✅ Network ports 18082, 18083, 18084 available

Step 1: Obtain the Management Pack

Download the Spark3 management pack:

The mpack includes all three Spark versions ambari-mpacks-odp-5566-spark3-3.5.5.3.2.3.5-2.tar.gz

Step 2: Install the Management Pack

On your Ambari Server, run:

Install the mpack ambari-server install-mpack \\ --mpack=._ambari-mpacks-odp-5566-spark3-3.5.5.3.2.3.5-2.tar.gz \\ --verbose # Restart Ambari Server ambari-server restart

Expected Output:

INFO: Installing management pack INFO: Loading properties from _etc_ambari-server_conf_ambari.properties INFO: Management pack installed successfully

Step 3: Add Spark3 Services via Ambari UI

Adding Spark3 3.5.5 (Recommended)

1. Log in to Ambari Web UI

   1. Navigate to http://<ambari-server>:8080
   2. Log in with admin credentials

2. Add Service

   1. Click "Actions" → "Add Service"
   2. Select "Spark3 3.5.5" from the service list
   3. Click "Next"

3. Assign Components

   • History Server: Select host(s) for Spark History Server
   • Thrift Server: (Optional) Select host(s) for Thrift Server
   • Client: Select hosts that need Spark client libraries
   • Click "Next"

4. Configure Service: Review default configurations. Key settings:

   1. History Server Port: 18082
   2. Log Directory: /var/log/spark3
   3. PID Directory: /var/run/spark3
   4. Click "Next"

5. Deploy Service

   1. Review the deployment plan
   2. Click "Deploy"
   3. Monitor installation progress
   4. Click "Complete" when finished

Adding Spark3 3.3.3 or 3.5.1

Repeat the above steps, selecting: Spark 3.5.1 or Spark 3.3.3

Step 4: Verify Installation

Check Services in Ambari UI

Navigate to the Ambari dashboard and verify:

• ✅ All installed Spark3 services show "Started" status
• ✅ All components are healthy (green indicators)

Verify History Server UIs

Access each History Server:

Spark 3.5.5 http:__<host>:18082 # Spark 3.3.3 http:__<host>:18083 # Spark 3.5.1 http:__<host>:18084

Each UI should display the correct Spark version.

Verify Client Installation

On client nodes:

# Check installed packages

rpm -qa | grep spark3

​

# Verify symlinks

ls -la _usr_odp_current_ | grep spark3

​

# Test Spark Shell (3.5.5)

_usr_odp_current_spark3-client_bin_spark-shell --version

​

# Test Spark Shell (3.3.3)

_usr_odp_current_spark3_3_3_3-client_bin_spark-shell --version

​

# Test Spark Shell (3.5.1)

_usr_odp_current_spark3_3_5_1-client_bin_spark-shell --version