Install Multiple Livy 3

Prerequisites

Before installing multiple Livy3 versions, ensure:

• ✅ Ambari Server 2.7.8.4-2/3

• ✅ ODP 3.0+ stack installed

• ✅ Required dependencies: HDFS, YARN, HIVE

• ✅ Corresponding Spark3 versions installed

  • Livy 3.5.5 requires Spark 3.5.5

  • Livy 3.3.3 requires Spark 3.3.3

  • Livy 3.5.1 requires Spark 3.5.1

• ✅ Network ports available:

  • Port 8355 for Livy 3.5.5 (recommended)

  • Port 8351 for Livy 3.5.1

  • Port 8333 for Livy 3.3.3

• ✅ Sufficient cluster resources for multiple Livy servers

Step 1: Obtain the Management Pack

The Livy management packs are bundled with the corresponding Spark3 mpacks:

Step 2: Install the Management Packs

On your Ambari Server, run:

Expected Output:

Step 3: Add Spark3 Services with Livy via Ambari UI

Adding Livy3 3.5.5 (Recommended)

1. Log in to Ambari Web UI

   • Navigate to http://<ambari-server>:8080

   • Log in with admin credentials

2. Add Service

   • Click "Actions" → "Add Service"

   • Select "Spark3 3.5.5" from the service list

   • Click "Next"

3. Assign Components

   • History Server: Select host(s) for Spark History Server

   • Livy Server: ✅ Enable and select host(s) for Livy3 Server

   • Thrift Server: (Optional) Select host(s) for Thrift Server

   • Client: Select hosts that need Spark client libraries

   • Click "Next"

4. Configure Service Review default configurations. Key Livy settings:

   • Livy Server Port: 8355 (default for 3.5.5)

   • Log Directory: /var/log/livy3

   • PID Directory: /var/run/livy3

   • Spark Home: /usr/odp/current/spark3-client

   • Click "Next"

5. Deploy Service

   • Review deployment plan

   • Click "Deploy"

   • Monitor installation progress

   • Click "Complete" when finished

Adding Livy3 3.3.3 or 3.5.1

Repeat the above steps, selecting:

• Spark 3.3.3 (includes Livy 3.3.3)

• Spark 3.5.1 (includes Livy 3.5.1)

Step 4: Configure Port Allocation

To avoid port conflicts, configure unique ports for each Livy instance:

Configure via Ambari UI:

1. Navigate to Services → SPARK3_3_3_3 (or respective version)

2. Click "Configs" tab

3. Find "Advanced livy3-3.3.3-conf" section

4. Set livy.server.port = 8333

5. Save changes

6. Restart Livy Server

Step 5: Verify Installation

Check Services in Ambari UI

Navigate to the Ambari dashboard and verify:

• ✅ All installed Livy3 services show "Started" status

• ✅ All components are healthy (green indicators)

• ✅ No port conflicts reported

Verify Livy REST API Endpoints

Access each Livy server's REST API:

Each endpoint should return the correct Livy version information:

Verify Client Installation

On client nodes: