Migrate a Java application using Maven

This walkthrough demonstrates how to migrate a Java application from Spark 2.4.8 and JDK 8 to Spark 3.5.5 and JDK 17 using Spark Migrate CLI.

The example uses java-order-pipeline, a daily order processing application built with Maven. It uses Lombok, Jackson, and JAXB libraries and includes six JUnit tests.

The migration updates incompatible dependencies, replaces unsupported Spark APIs, and configures the build and test environment.

1. Run the migration

Navigate to the working directory and run the migration:

cd ~/work
spark-migrate-cli run -r java-order-pipeline --job-id order-pipeline-java

[1/2] analyze  ~/work/java-order-pipeline
  [   0s] a Java repo, built with maven
  [   3s] OpenRewrite: 13 Java file(s), with the repo's classpath
analyze: SUCCESS

Migratable automatically
  0 must fix · 9 done for you

[2/2] codegen  rewrite Tier 1, inject Tier 2 config, then compile and run the test suite
code_changes: SUCCESS
compile: SUCCESS
unit_testing: SUCCESS
  tests: total=6 passed=6 failed=0 skipped=0

The migrated application compiled successfully using Spark 3.5.5 and JDK 17, and all six unit tests passed.

2. Review the changes

Open the generated changes.html report to review the modifications and their associated migration rules.

The following sections describe the key changes.

Dependency upgrades

The tool upgrades dependencies that are incompatible with Spark 3.5.5 or JDK 17.

The tool retains compatible dependencies, such as Guava 32 and test-scope JUnit, without downgrading their versions.

Build configuration

The tool updates the Maven build configuration to use Spark 3.5.5 and Scala 2.12.

It also configures the Maven Surefire argLine with the required JDK 17 JVM flags and Spark compatibility settings so that unit tests run with the appropriate configuration.

Application code

The tool replaces unsupported Spark APIs with their supported equivalents.

For example, unionAll is replaced with union:

-        return today.unionAll(backfill);
+        return today.union(backfill);

-                metrics.shuffleBytesWritten(), metrics.shuffleWriteTime(), metrics.shuffleRecordsWritten());
+                metrics.bytesWritten(), metrics.writeTime(), metrics.recordsWritten());

Compatibility configuration

The tool generates the required Spark compatibility configurations.

The changes.html report lists each injected configuration, its associated finding, and the source line that requires it.

Review these configurations to understand the compatibility settings applied during migration.

3. Apply the changes

After reviewing the migration report, apply the changes to a new Git branch:

spark-migrate-cli apply --job-id order-pipeline-java --branch spark-3.5-migration

4. Configure and build the application

Generate the Spark configuration flags required to launch the migrated application:

spark-migrate-cli conf --job-id order-pipeline-java

5. Validate the migration

Before deploying the migrated application to production:

1. Run the original Spark 2.4.8 application and the migrated Spark 3.5.5 application using the same input data.
2. Compare the outputs to identify any differences in processing results.
3. Investigate and resolve unexpected differences before cutover.

For detailed instructions, see Validate the data.

Result

The java-order-pipeline application was successfully migrated from Spark 2.4.8 and JDK 8 to Spark 3.5.5 and JDK 17.

The tool automatically upgraded incompatible dependencies, updated unsupported APIs, and configured the build and test environment. All six unit tests passed.

The remaining manual configuration step is to add the generated Spark compatibility settings to the application's launch configuration. Production readiness remains subject to output validation and deployment checks.