Install Multiple Spark 3 Versions

Prerequisites

Before installing multiple Spark3 versions, ensure:

• ✅ Ambari Server 2.7.0 or higher

• ✅ ODP 3.0+ stack installed

• ✅ Required dependencies: HDFS, YARN, HIVE

• ✅ Sufficient cluster resources for multiple History Servers

• ✅ Network ports 18082, 18083, 18084 available

Step 1: Obtain the Management Pack

Download the Spark3 management pack:

Step 2: Install the Management Pack

On your Ambari Server, run:

Expected Output:

Step 3: Add Spark3 Services via Ambari UI

Adding Spark3 3.5.5 (Recommended)

1. Log in to Ambari Web UI

   1. Navigate to http://<ambari-server>:8080

   2. Log in with admin credentials

2. Add Service

   1. Click "Actions" → "Add Service"

   2. Select "Spark3 3.5.5" from the service list

   3. Click "Next"

3. Assign Components

   • History Server: Select host(s) for Spark History Server

   • Thrift Server: (Optional) Select host(s) for Thrift Server

   • Client: Select hosts that need Spark client libraries

   • Click "Next"

4. Configure Service: Review default configurations. Key settings:

   1. History Server Port: 18082

   2. Log Directory: /var/log/spark3

   3. PID Directory: /var/run/spark3

   4. Click "Next"

5. Deploy Service

   1. Review the deployment plan

   2. Click "Deploy"

   3. Monitor installation progress

   4. Click "Complete" when finished

Adding Spark3 3.3.3 or 3.5.1

Repeat the above steps, selecting: Spark 3.5.1 or Spark 3.3.3

Step 4: Verify Installation

Check Services in Ambari UI

Navigate to the Ambari dashboard and verify:

• ✅ All installed Spark3 services show "Started" status

• ✅ All components are healthy (green indicators)

Verify History Server UIs

Access each History Server:

Each UI should display the correct Spark version.

Verify Client Installation

On client nodes: