Livy 3 Limitations

Independent Livy Instances ⚠️

Impact:

• Each Livy version runs as a separate process
• No session sharing between Livy instances
• Client applications must connect to the correct endpoint

Consideration:

• Plan client routing carefully
• Use load balancers if needed for each version
• Document endpoint URLs for teams

Port Management

Impact:

• Each Livy instance requires a unique port
• Firewall rules must allow access to all ports
• Client applications need port-specific configuration

Recommended Port Allocation:

Spark Version Coupling

Impact:

• Each Livy version should use its corresponding Spark version
• Mixing versions can cause compatibility issues
• The configuration must point to the correct Spark installation

Best Practice:

# Ensure SPARK_HOME matches Livy version

# For Livy 3.3.3:

SPARK_HOME=/usr/odp/current/spark3_3_3_3-client

​

# For Livy 3.5.5:

SPARK_HOME=/usr/odp/current/spark3-client

API Compatibility

Impact:

• Different Livy versions may have API differences
• Client applications may need version-specific code
• Behavioral changes between versions

Mitigation:

• Review Livy release notes for API changes
• Test client applications against the target Livy version
• Maintain version-specific client libraries if needed