Get started

Get started with Spark Migrate CLI to analyze and migrate your existing Spark applications to Spark 3.5.5.

Prerequisites

You need	For
JDK 17	the compile and test steps (point JAVA_HOME at it)
git	creating and applying the patch
sbt	Scala repositories built with sbt
Maven and coursier (cs)	Maven repositories, Scala or Java
Python 3.10–3.13	the analysis engine for PySpark and embedded SQL; your job keeps its own Python
Maven Central, or your mirror	resolving Spark 3.5.5 and upgraded libraries through your own sbt/Maven settings

The bundled Python engine supports Linux x86_64, Linux aarch64 and Apple Silicon macOS. Install only what your repositories need.

Install

sha256sum -c spark-migrate-cli-1.1.0.tar.gz.sha256        # macOS: shasum -a 256 -c ...
tar -xzf spark-migrate-cli-1.1.0.tar.gz
spark-migrate-cli-1.1.0/install.sh --profile ~/.profile
source ~/.profile

Check the machine

spark-migrate-cli doctor

Tools
  ok      git                 git version 2.50.1
  ok      java                openjdk version "17.0.19" 2026-04-21 LTS
  ok      sbt                 2.0.4 (launcher)
  ok      mvn                 Apache Maven 3.9.16
  ok      cs                  2.1.24
  ...
What this machine can migrate
  Scala + sbt    READY
  Scala + Maven  READY
  Java + Maven   READY
  PySpark        READY

Try the built-in sample (5 minutes)

mkdir ~/sm-try && cd ~/sm-try
spark-migrate-cli demo --yes                 # unpack a Spark 2.4.8 sbt job, migrate it, open the report
spark-migrate-cli apply --job-id demo
grep -n "sparkVersion\|scalaVersion" spark-migrate-demo-repo/build.sbt

4:scalaVersion := "2.12.18"
6:val sparkVersion = "3.5.5"
9:  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
10:  "org.apache.spark" %% "spark-sql"  % sparkVersion % "provided",

Migrate your own repository

spark-migrate-cli run -r ./my-spark2-repo --job-id my-job