Livy 3 FAQs

Q: Can I install only one or two Livy versions?

A: Yes! You can install any combination of the three versions based on your requirements. Installation is independent and modular. Each Livy version comes with its corresponding Spark service installation.

Q: Which Livy version should I use for new applications?

A: Livy 3.5.5 is recommended for all new applications. It provides:

• Latest bug fixes and features
• Best compatibility with Spark 3.5.5
• Active maintenance and updates
• Improved performance and stability

Q: Can I run multiple sessions from the same Livy instance?

A: Yes! Each Livy instance supports multiple concurrent sessions. The limit depends on:

• Available cluster resources (YARN capacity)
• Livy server memory configuration
• Network bandwidth
• Typical range: 10-100+ concurrent sessions per Livy server

Q: Do I need separate YARN queues for different Livy versions?

A: Not required, but recommended for:

• Resource isolation between versions
• Capacity planning per version
• Cost allocation tracking
• Performance monitoring and troubleshooting

Configure queue per session:

curl -X POST http://<host>:8333/sessions \

  -d '{"kind":"spark","conf":{"spark.yarn.queue":"livy_3_3_3_queue"}}'

Q: How do I migrate applications from one Livy version to another?

A: Follow this migration checklist:

1. Test application on the new Livy version
2. Update client endpoint URLs
3. Deploy the updated application
4. Monitor for errors or behavior changes
5. Rollback to the old version if issues arise
6. Decommission the old version after the validation period

Q: Can different Livy versions share the same Spark installation?

A: Not recommended. Each Livy version should use its corresponding Spark version:

• Livy 3.5.5 → Spark 3.5.5
• Livy 3.3.3 → Spark 3.3.3
• Livy 3.5.1 → Spark 3.5.1

Mixing versions can cause:

• Classpath conflicts
• API incompatibilities
• Unpredictable behavior

Q: What happens if I restart the Livy server with active sessions?

A: Active sessions will be terminated:

• Interactive sessions: Lost (cannot be recovered)
• Batch jobs: Continue running in YARN (can be monitored via YARN UI)

Best Practice: Drain sessions before restart:

# Check active sessions

curl http://<host>:8333/sessions

​

# Wait for sessions to complete or delete gracefully

curl -X DELETE http://<host>:8333/sessions/{session-id}

Q: Can I load balance across multiple Livy instances of the same version?

A: Yes! For high availability:

1. Deploy multiple Livy servers (same version, different hosts)
2. Use load balancer (HAProxy, Nginx, F5)
3. Configure health checks on /version endpoint

Example Nginx configuration:

upstream livy_355_cluster {

    server livy-host1:8355 max_fails=3 fail_timeout=30s;

    server livy-host2:8355 max_fails=3 fail_timeout=30s;

}

​

server {

    listen 80;

    location / {

        proxy_pass http://livy_355_cluster;

    }

}

Q: How do I secure communication between the client and Livy?

A: Multiple security layers:

1. Network Security:

   • Use firewall rules to restrict access
   • VPN for remote access

2. Authentication:

livy.server.auth.type = kerberos

3. SSL/TLS:

livy.keystore = /path/to/keystore.jks

4. Authorization:

livy.server.access-control.enabled = true

livy.server.access-control.users = authorized_users