Spark 4 Connect

Spark 4.1 Connect is available with ODP 3.3.6.5. Spark 4 Connect differs from Spark 3.5.5 Connect in its default port, client installation, Scala version, configuration paths, and Ambari configuration.

For information about Spark 3.5.5 Connect, see Spark Connect.

Spark 4 Connect configuration

The following table compares the Spark 3.5.5 and Spark 4.1.1 Connect configurations.

Configuration

Spark 3.5.5

Spark 4.1.1

Server component

Spark3 Connect

SPARK4_CONNECT_SERVER under the Spark4 service

Default gRPC port

15002

15005

Default bind address

localhost

Loopback when the binding address is empty

Scala binary version

2.12

2.13

Mpack configuration file

spark3-connect.xml

spark4-connect.xml

Host configuration file

/etc/spark3/conf/spark-connect.conf

/etc/spark4/conf/spark-connect.conf

Ambari configuration

Spark3 > Advanced spark-connect

Spark4 > Advanced spark4-connect

Log directory

/var/log/spark3/

/var/log/spark4/

Server log prefix

spark-*-SparkConnectServer-*.out

spark-*-SparkConnectServer-*.out

Install the Spark 4 Connect client

For Spark 4.1.1, don't install the Connect client by using pip install pyspark[connect]==4.1.1. This command downloads the full PySpark package, including the JVM JAR files.

Instead, install the lightweight PySpark client and Spark Connect packages provided for ODP.

Prerequisites

Ensure that Python 3.9 or later is installed on the client system.

Install the Python client

  1. Create and activate a Python virtual environment:

    python3 -m venv ~/spark4-connect source ~/spark4-connect/bin/activate pip install --upgrade pip

  2. Install the PySpark client package:

    pip install https://mirror-stg.odp.acceldata.dev/staging/OCR-2426/pyspark-client-4.1.1.tar.gz

  3. Install the Spark Connect package:

    pip install https://mirror-stg.odp.acceldata.dev/staging/OCR-2426/pyspark-connect-4.1.1.tar.gz

  4. Verify the installation:

    python -c "import pyspark; print(pyspark.__version__)"

    The command returns:

    4.1.1

The combined installation requires approximately 30 MB, including dependencies such as NumPy, pandas, PyArrow, and gRPC.

Configure a Scala client

Spark 4.1.1 uses Scala 2.13. For Maven-based applications, add the following dependency:

<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-connect-client-jvm_2.13</artifactId> <version>4.1.1</version> </dependency>

Connect to the Spark 4 Connect server

The default Spark 4 Connect gRPC port is 15005.

Python

Use the following example to connect from a Python client:

from pyspark.sql import SparkSession spark = SparkSession.builder.remote( "sc://<connect-host>:15005" ).getOrCreate() print(spark.range(1000).count()) spark.stop()

Scala

Use the following example to connect from a Scala client:

val spark = SparkSession.builder() .remote("sc://<connect-host>:15005") .getOrCreate()

Important: Spark 4 Connect uses port 15005. Spark 3.5.5 Connect uses port 15002.

Use distributed storage for file operations

Use a distributed file system, such as HDFS, S3, ABFS, or GCS, for files accessed by Spark Connect workloads.

Avoid using file:// paths for distributed Spark workloads. Local file paths are resolved on the host where the Spark process or executor accesses the file and might not refer to the client system.

For example, use:

hdfs:///path/to/data s3a://bucket/path/to/data abfs://container@account/path/to/data

instead of:

file:///tmp/data

Known issues and limitations

The following issues apply to Spark 4.1.1 with the ODP 3.3.6.5 Mpack.

Spark Connect binds to the loopback address by default

The spark4-connect configuration includes the following default settings:

spark.connect.grpc.binding.address= spark.connect.grpc.binding.port=15005

When spark.connect.grpc.binding.address is empty, the Spark Connect server binds to the loopback address. As a result, remote clients can't connect to the server.

Configure remote client access

To allow remote clients to connect:

  1. In Ambari, go to Services > Spark4 > Configs.

  2. Expand Advanced spark4-connect.

  3. Set spark.connect.grpc.binding.address to:

    0.0.0.0

  4. Save the configuration.

  5. Restart the Spark Connect Server component.

Security: Binding the service to 0.0.0.0 makes it reachable on all available network interfaces. Ensure that network access to port 15005 is restricted to trusted clients.

Use an SSH tunnel

If you don't want to expose the Spark Connect server on a network interface, create an SSH tunnel from the client:

ssh -i ~/path/to/key.pem -N \ -L 15005:127.0.0.1:15005 acceldata@<connect-host>

Then connect to:

sc://localhost:15005

An empty authentication token can cause authentication failures

In the Spark 4.1.1 build included with ODP 3.3.6.5, the Spark Connect authentication interceptor can be enabled when spark.connect.authenticate.token is present, even when the property has an empty value.

Clients can then fail with the following error:

StatusCode.UNAUTHENTICATED details = "No authentication token provided"

Use one of the following configurations.

Disable token authentication

If token authentication isn't required, remove the spark.connect.authenticate.token property from Advanced spark4-connect in Ambari, and then restart the Spark Connect Server.

Alternatively, configure spark.connect.grpc.interceptor.classes so that the authentication interceptor isn't loaded.

Configure token authentication

If authentication is required:

  1. Set spark.connect.authenticate.token to a nonempty token.

  2. Restart the Spark Connect Server.

  3. Configure clients to provide the token:

    spark = SparkSession.builder.remote( "sc://<connect-host>:15005/;token=<token>" ).getOrCreate()

Important: PySpark 4.1 automatically enables TLS on the client when a token is included in the connection URL. Configure TLS on the Spark Connect server before using token authentication. Otherwise, the TLS handshake can fail.

spark-connect-shell requires an interactive terminal

The following command wraps the Ammonite REPL:

/usr/odp/current/spark4-client/bin/spark-connect-shell

The shell requires an interactive terminal. Noninteractive methods such as piping a heredoc, using -i with a Scala script, or running the shell with nohup aren't supported.

For noninteractive Scala Spark Connect workloads, build an application with the following client dependency:

spark-connect-client-jvm_2.13

Then run the compiled application as a standard Scala application.

--remote conflicts with the configured Spark master

The Spark4 client configuration can define:

spark.master yarn

If you run the following command while spark.master is defined:

spark-connect-shell --remote sc://<connect-host>:15005

Spark returns the following error:

SparkException: Remote cannot be specified with master and/or deploy mode.

As a workaround, set SPARK_CONF_DIR to a configuration directory whose spark-defaults.conf doesn't define spark.master, and then start spark-connect-shell.

Parquet writes can fail

Parquet writes in Spark 4.1.1 with the ODP 3.3.6.5 build can fail with the following error:

NoSuchMethodError: shaded.parquet.it.unimi.dsi.fastutil.longs.LongList.of()

This issue affects Spark 4.1.1 generally and isn't specific to Spark Connect.

ORC, CSV, and JSON writes are supported. Until the Parquet issue is resolved, use ORC or another supported format when possible.

The Ambari-managed Spark Connect server starts in local mode by default. Therefore, processing doesn't scale beyond the server host.

To use YARN for Spark Connect workloads:

  1. Ensure that the spark user has permission to submit applications to the required YARN queue.

  2. Configure the Spark Connect server to use:

    --master yarn --deploy-mode client

  3. Restart the Spark Connect Server.

Note: YARN queue permissions might need to be configured in Ranger before the Spark Connect server can submit applications.

Spark 4 Connect configuration reference

Configure Spark 4 Connect properties in Spark4 > Configs > Advanced spark4-connect.

Property

Recommended value

Description

spark.connect.grpc.binding.address

0.0.0.0 for remote access

Allows remote clients to connect to the Spark Connect server.

spark.connect.grpc.binding.port

15005

Specifies the Spark 4 Connect gRPC port.

spark.connect.authenticate.token

Remove the property when authentication isn't required, or specify a nonempty token and configure TLS

Controls Spark Connect token authentication.

spark.authenticate

false

Retains the default setting when Spark authentication isn't required.

Spark Connect version compatibility

Spark version

Python client

Scala client artifact

Default port

ODP configuration

Spark 3.5.5

pip install pyspark[connect]==3.5.5

spark-connect-client-jvm_2.12:3.5.5

15002

spark-connect.conf

Spark 4.1.1

pyspark-client-4.1.1 and pyspark-connect-4.1.1 packages

spark-connect-client-jvm_2.13:4.1.1

15005

spark4-connect.conf

Validation

Spark 4 Connect was validated with ODP 3.3.6.5 for the following scenarios:

  • PySpark Connect from an ODP cluster host.

  • PySpark Connect from a remote client to the cluster.

  • Cross-client access to data written from the cluster.

  • ORC, CSV, and JSON read and write operations.

Related information

For Spark 3.5.5 Connect configuration and usage, see Spark Connect.


  Last updated on Aug 27, 2026
Next to read:
Install Hue
null
On This Page
Spark 4 ConnectSpark 4 Connect configurationInstall the Spark 4 Connect clientConfigure a Scala clientConnect to the Spark 4 Connect serverUse distributed storage for file operationsKnown issues and limitationsSpark 4 Connect configuration referenceSpark Connect version compatibilityValidationRelated informationPrerequisitesInstall the Python clientPythonScalaSpark Connect binds to the loopback address by defaultAn empty authentication token can cause authentication failuresspark-connect-shell requires an interactive terminal--remote conflicts with the configured Spark masterParquet writes can failSpark Connect server runs in local mode by default