Livy 3 Limitations

Independent Livy Instances ⚠️

Impact:

Each Livy version runs as a separate process
No session sharing between Livy instances
Client applications must connect to the correct endpoint

Consideration:

Plan client routing carefully
Use load balancers if needed for each version
Document endpoint URLs for teams

Port Management

Impact:

Each Livy instance requires a unique port
Firewall rules must allow access to all ports
Client applications need port-specific configuration

Recommended Port Allocation:

Version	Port	Purpose
3.5.5	8355	Default/Production
3.5.1	8351	Compatibility Testing
3.3.3	8333	Legacy Support

Spark Version Coupling

Impact:

Each Livy version should use its corresponding Spark version
Mixing versions can cause compatibility issues
The configuration must point to the correct Spark installation

Best Practice:

# Ensure SPARK_HOME matches Livy version # For Livy 3.3.3: SPARK_HOME=/usr/odp/current/spark3_3_3_3-client # For Livy 3.5.5: SPARK_HOME=/usr/odp/current/spark3-client

API Compatibility

Impact:

Different Livy versions may have API differences
Client applications may need version-specific code
Behavioral changes between versions

Mitigation:

Review Livy release notes for API changes
Test client applications against the target Livy version
Maintain version-specific client libraries if needed

Python2 Compatibility in Spark 3 with Livy3

Impact:

Spark 3’s PySpark code requires Python 3.
If Livy initializes a Spark 3 session using Python 2, a SyntaxError: invalid syntax occurs.
Spark session initialization fails because Python 2 cannot parse Python 3–only language features (e.g., type annotations like Union[str, float]).

Error with Python 2

ERROR PythonInterpreter: Process has died with 1 ERROR PythonInterpreter: Traceback (most recent call last): File "/tmp/9175131608716314470", line 726, in <module> sys.exit(main()) File "/tmp/9175131608716314470", line 570, in main exec('from pyspark.sql import HiveContext', global_dict) File "<string>", line 1, in <module> File "/usr/odp/current/spark3-client/python/lib/pyspark.zip/pyspark/__init__.py", line 75 def since(version: Union[str, float]) -> Callable[[_F], _F]: ^ SyntaxError: invalid syntax

Mitigation / Workaround:

Configure Livy to use Python 3 by adding the following to livy3-env

export PYSPARKPYTHON=/usr/bin/python3 export PYSPARK DRIVER_PYTHON=/usr/bin/python3

Restart the Livy service after applying these changes.
Ensures both Spark driver and executors run Python 3, allowing Spark 3 sessions to start successfully.

Last updated on Jan 29, 2026

Was this page helpful?