Spark Migrate CLI overview

Spark Migrate CLI is a command-line tool that helps migrate existing Apache Spark 2.x applications written in Scala, Java, or PySpark to Apache Spark 3.5.5.

The tool identifies compatibility issues, applies supported code and configuration changes, updates dependencies, and compiles and tests the migrated application.

This page explains the challenges involved in upgrading Spark applications, how Spark Migrate CLI addresses them, and the supported migration workflows.

Target versions

The current migration target is Apache Spark 3.5.5 with Scala 2.12 and JDK 17.

The problem: A Spark upgrade is not just a version change

Upgrading from Spark 2.4.8 to Spark 3.5.5 involves more than updating the Spark version in a build file.

Spark 3 introduces changes to programming languages, JDK requirements, libraries, APIs, and runtime behavior. Some changes cause compilation or runtime failures, while others affect application results without generating errors.

These compatibility issues fall into six areas.

1. Platform compatibility

Spark 3.5 introduces changes to the underlying platform and its dependencies.

2. Code that no longer compiles

Spark 3 removes or changes several APIs supported in Spark 2.4.

Examples include:

• Accumulator v1 API (sc.accumulator).
• HiveContext and the SQLContext constructor.
• ProcessingTime.
• ShuffleWriteMetrics accessors.
• MLlib APIs such as KMeans.train(..., runs, ...) and the *WithSGD family.
• ExpressionEncoder.toRow.
• Koalas APIs, now available through pyspark.pandas.

These issues typically cause compilation failures or compatibility errors.

Spark Migrate CLI automatically rewrites APIs that have a supported Spark 3 equivalent. For issues that cannot be resolved automatically, the tool reports the affected source location and required action.

3. Code that compiles but fails at runtime

Some applications compile successfully but fail when processing data on Spark 3.5.

Common examples include:

• Datetime parsing: Spark 3 uses stricter date and timestamp parsing. Values accepted by Spark 2.4 may produce SparkUpgradeException. Certain date pattern characters, including Y, w, u, and E, may also cause errors.
• Table writes: Spark 3 may reject incompatible values during table writes where Spark 2.4 previously produced null values.
• Historical dates: Reading or writing older dates in Parquet and Avro may cause compatibility errors. Applications using sentinel dates such as 0001-01-01 require additional attention.
• Self-joins: References to columns in self-joins may fail because Spark 3 detects ambiguous column references that Spark 2.4 previously accepted.

4. Code that runs but produces different results

Some Spark 3 behavior changes do not cause compilation or runtime errors. Applications may complete successfully while producing different results.

Examples include:

These differences may remain undetected unless tests explicitly verify the affected values.

5. Dependencies and connectors

Libraries and connectors that depend on Spark may require new versions when migrating to Spark 3.5.

For example:

• Delta Lake 0.6 requires migration to a compatible delta-spark 3.x version and additional session configurations.
• spark-xml and spark-avro require compatible Spark 3 artifacts.
• hadoop-aws dependencies may need to move from Hadoop 2.7 to Hadoop 3.3.
• Older Jackson and Guava versions may conflict with the libraries included in Spark 3.5.

Spark dependencies must also be configured correctly. Including Spark dependencies in the application JAR instead of marking them as provided may cause classpath conflicts.

6. Build configurations and launch scripts

Spark migrations may also require changes to build tools and application launch configurations.

Examples include:

• spark-submit scripts referencing packages built for Scala 2.11.
• SPARK_HOME pointing to a Spark 2 installation.
• Launch commands using the removed yarn-client master URL.
• Scripts invoking vendor-specific commands such as spark2-submit.
• Older sbt and Maven plugins that are incompatible with JDK 17.

Why Spark migrations are challenging

Spark migration issues can be difficult to identify and resolve for several reasons:

• Issues occur across multiple files. Compatibility problems may exist in application code, embedded SQL, standalone SQL files, build configurations, dependency definitions, and launch scripts.
• Some changes are difficult to detect. Applications may run successfully but produce different results because of changes in Spark behavior.
• Legacy configurations require careful selection. Enabling unnecessary spark.sql.legacy.* settings can preserve outdated behavior and make future cleanup difficult.
• Migration requires repeated effort. Organizations with multiple Spark applications must identify and resolve similar compatibility issues across repositories.

What Spark Migrate CLI does for you

Spark Migrate CLI automates supported migration tasks and identifies issues that require manual intervention.

The tool uses 145 documented migration rules, each identified by a SPARK-MIG code.

Each rule explains:

• What changed and the Spark version in which the change occurred.
• When the rule applies.
• What the tool changes automatically.
• Whether manual intervention is required.

Every identified issue has one of three outcomes:

• Rewritten: The tool automatically modifies the affected code.
• Restored: The tool adds a compatibility configuration to preserve existing behavior.
• Reported: The tool identifies the issue and recommends a resolution when an automatic change is not safe.

What you get

Spark Migrate CLI provides the following outputs:

• Analysis report: Identifies compatibility issues, affected source locations, and required actions before code generation.
• Reviewable changes: Generates changes to application code, build files, dependencies, configurations, and supported launch scripts.
• Build and test results: Compiles the migrated application and runs its existing tests against Spark 3.5.5.
• Change explanations: Associates each code change and configuration with the migration rule that required it.
• Data validation: Compares the original and migrated application outputs to identify differences.
• Git integration: Applies changes to a new Git branch for review without modifying the original repository until requested.

The analysis report

Spark Migrate CLI generates an HTML analysis report before modifying application code.

The report provides an overall migration verdict and lists compatibility issues, affected source locations, and recommended actions.

Migration verdicts

Finding categories

The analysis report groups findings into the following categories:

Each finding includes:

• The migration rule and a link to its documentation.
• An explanation of the Spark compatibility change.
• The required action, if any.
• The affected source file, line number, and proposed modification.

You can search and filter the report and attach it to a ticket or pull request.

What it saves

The following table summarizes the migration results from the example applications.

In these examples, Spark Migrate CLI handled the remaining supported migration tasks, including:

• Dependency and connector upgrades.
• spark-submit script updates.
• Legacy compatibility configurations.
• Supported MLlib and Koalas API changes.
• Compilation and unit testing.

Actual migration time depends on the repository, dependencies, and build and test duration.

How it works

Spark Migrate CLI processes each migration as a job.

Each job represents one repository and is stored in a dedicated directory:

./spark-migrate-jobs/<job-id>/

1. Analyze

The analyze command creates a copy of the source repository and uses the appropriate analysis engines.

The tool generates the analysis report containing compatibility findings and recommended actions.

2. Generate code changes

The codegen command creates a separate working copy and:

• Rewrites supported code to preserve Spark 2.4 behavior.
• Adds compatibility configurations where required.
• Updates the build to Spark 3.5.5, Scala 2.12.18, and JDK 17, where applicable.

3. Compile and test

The tool compiles the migrated application and runs the repository's existing tests.

Compilation or test failures are reported for investigation and resolution.

4. Validate

The validate command compares the outputs of the original Spark 2.x application and the migrated Spark 3.5.5 application.

Validation checks include:

• Schema comparison.
• Row counts.
• Row-level differences based on key columns.
• Aggregate values.

By default, failed validation prevents the migration changes from being applied.

5. Apply

The apply command applies the generated changes to the repository on a new Git branch without committing them.

Alternatively, the migrate command processes a remote Git repository, commits the migration changes to a new branch, and optionally pushes the branch.

What it supports

Spark Migrate CLI supports the following languages and build systems.

Supported source and target versions

Multi-module sbt and Maven builds are supported. Each migration job processes one repository.