Connect to a Spark Connect Server from a Laptop

You can connect to a Spark 4.1 Connect server running on an ODP cluster from your laptop without installing a JVM. For a Python client, you need only Python and pip.

For information about Spark Connect, see Spark Connect Overview 4.1.1.

Prerequisites

Before you connect to the Spark Connect server, verify the following requirements:

Python 3.9 or later. This configuration is validated with Python 3.11 and 3.12.
Network connectivity from your laptop to the Spark Connect server on port 15005.
The hostname or IP address of the Spark Connect server.

For example, the server used in this procedure is:

10.101.11.114:15005

If the Spark Connect server is bound only to the loopback address (127.0.0.1), configure an SSH tunnel before connecting. For more information, see Troubleshooting.

Install the Python Client

The full PySpark package is approximately 1.3 GB because it includes the JVM JAR files required to run Spark locally.

For a Spark Connect-only client, you don't need the full PySpark distribution. Install the lightweight client packages instead.

Create a Python virtual environment:

python3 -m venv ~/spark4-connect

Activate the virtual environment:

source ~/spark4-connect/bin/activate

Upgrade pip:

pip install --upgrade pip

Install the PySpark client package:

pip install https://mirror-stg.odp.acceldata.dev/staging/OCR-2426/pyspark-client-4.1.1.tar.gz

Install the Spark Connect package:

pip install https://mirror-stg.odp.acceldata.dev/staging/OCR-2426/pyspark-connect-4.1.1.tar.gz

Verify the installed PySpark version:

python -c "import pyspark; print(pyspark.__version__)"

Expected output:

4.1.1

The complete client installation is approximately 30 MB, including NumPy, pandas, PyArrow, gRPC, and the Spark client packages.

Don't install pyspark-4.1.1.tar.gz on the client system unless you need a full local Spark installation. The package is approximately 1.3 GB and includes the JVM libraries required to run Spark locally.

Connect Using Python

Use SparkSession.builder.remote() to connect to the Spark Connect server.

For example:

from pyspark.sql import SparkSession spark = SparkSession.builder.remote( "sc://10.101.11.114:15005" ).getOrCreate() print("count:", spark.range(1000).count()) df = spark.range(1_000_000).selectExpr( "id", "id % 100 as bucket" ) df.groupBy("bucket").count().orderBy("bucket").limit(5).show() spark.stop()

Save the code as demo.py.

Activate the virtual environment:

source ~/spark4-connect/bin/activate

Run the application:

python demo.py

Connect Using IntelliJ IDEA and Scala

To connect from a Scala application, create a Maven or Gradle project in IntelliJ IDEA.

For a Maven project, add the following dependency:

<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-connect-client-jvm_2.13</artifactId> <version>4.1.1</version> </dependency>

Use the following example to create a Spark Connect session:

import org.apache.spark.sql.SparkSession object Demo extends App { val spark = SparkSession.builder() .remote("sc://10.101.11.114:15005") .getOrCreate() println("count: " + spark.range(1000).count()) spark.stop() }

Configure the IntelliJ IDEA run configuration to use JVM 17.

No local Spark installation is required. The client communicates with the Spark Connect server by using gRPC over TCP.

Troubleshoot Connection Issues

Authentication Token Is Not Provided

You might receive the following error:

UNAUTHENTICATED: No authentication token provided

This error occurs when spark.connect.authenticate.token is configured with a nonempty value on the Spark Connect server.

To resolve the issue, use one of the following options:

Ask the cluster administrator to remove the spark.connect.authenticate.token value in Ambari > Services > Spark4 > Configs > Advanced spark4-connect, and then restart the Spark Connect server.
Obtain the configured token from the cluster administrator and include it in the connection URL.

For example:

spark = SparkSession.builder.remote( "sc://10.101.11.114:15005/;token=<token>" ).getOrCreate()

In PySpark 4.1, specifying a token in the connection URL automatically enables TLS on the client. The Spark Connect server must be configured with a compatible TLS certificate. If the server uses plaintext gRPC, remove the token or configure TLS on the server.

TLS Handshake Fails

You might receive an error similar to the following:

Handshake failed with error SSL_ERROR_SSL ... WRONG_VERSION_NUMBER

This error can occur when you specify ;token= in the connection URL. PySpark automatically enables TLS when a token is specified, but the server might be using plaintext gRPC.

To resolve the issue, either:

Remove the token from the connection URL if authentication isn't required.
Configure TLS on the Spark Connect server if token-based authentication is required.

Connection Is Refused

If the client can't establish a TCP connection, the Spark Connect server might be bound to 127.0.0.1 instead of 0.0.0.0.

Ask the cluster administrator to set the following property in Ambari and restart the Spark Connect server:

spark.connect.grpc.binding.address=0.0.0.0

Alternatively, create an SSH tunnel:

ssh -i ~/Cursor/git_pr/spark-test/us_dc_vast2.pem -N \ -L 15005:127.0.0.1:15005 acceldata@10.101.11.114

Then, connect to the local endpoint:

spark = SparkSession.builder.remote( "sc://localhost:15005" ).getOrCreate()

Access to Kerberos-Protected HDFS Data Fails

Spark Connect doesn't propagate Kerberos credentials from the client system to the Spark Connect server.

File operations run by using the identity of the user or principal that runs the Spark Connect server, typically the spark user.

To access HDFS data owned by another user, use one of the following approaches:

Ask the cluster administrator to run a dedicated Spark Connect server for the required user.
Configure the required HDFS or Ranger permissions for the Spark Connect server user.
Stage the required data by using a standard spark-submit application before accessing it through Spark Connect.

Choose a PySpark Package

Use the following table to determine which package to install:

Package	Approximate Size	When to Use
pyspark-4.1.1.tar.gz	1.36 GB	Use when you need a full local Spark installation, including JVM JAR files, to run spark-submit locally or use Spark local mode. You don't need this package for a Spark Connect-only client.
pyspark-client-4.1.1.tar.gz	1.6 MB	Provides the lightweight Python client APIs, including SparkSession and DataFrame APIs, without a local JVM. Required for the Spark Connect client.
pyspark-connect-4.1.1.tar.gz	4.7 KB	Provides the Spark Connect dependencies, including gRPC and Arrow support, and enables SparkSession.builder.remote(). Required for the Spark Connect client.

Validated Configuration

This configuration was validated on August 23, 2026, against Spark 4.1.1.3.3.6.5-1009 running as the Spark Connect server on an ODP 3.3.6.5 test cluster.

The following scenarios were validated:

On-cluster PySpark Connect with a 1,000-row count and groupBy operation.
PySpark Connect from a macOS laptop with a 100-row count and a 1-million-row groupBy operation.
ORC, CSV, and JSON file operations through Spark Connect.
Scala Ammonite shell (spark-connect-shell) in interactive mode.

Known Limitations

Parquet: Parquet operations fail because the Spark 4 assembly contains an issue with the shaded Parquet fastutil dependency. Use ORC with Spark 4 until the issue is resolved.
Scala Ammonite shell: spark-connect-shell requires an interactive TTY. Spark 4.1 doesn't support running the shell in scripted mode through a heredoc.

Last updated on Aug 27, 2026

Was this page helpful?