Migrate a Scala application using sbt

This walkthrough demonstrates how to migrate a Scala application from Spark 2.4.8 and Scala 2.11 to Spark 3.5.5 and Scala 2.12 using Spark Migrate CLI.

The example uses store-sales-etl, a nightly batch application that reads Parquet files, filters and joins data, aggregates daily sales totals, and runs an SQL report. The application uses a spark-submit script for execution and ScalaTest for unit testing.

The migration in this example completes without manual code changes.

1. Run the migration

Navigate to the working directory and run the migration:

cd ~/work
spark-migrate-cli run -r store-sales-etl --job-id store-sales

[1/2] analyze  ~/work/store-sales-etl
  [   0s] a Scala repo, built with sbt
  [   0s] Scalafix (sbt): compiling the repo for its types, then 48 rules over main and test sources
  [  24s] embedded SQL: linting with sqlfluff
analyze: SUCCESS

Migratable automatically
  0 must fix · 9 done for you · 1 FYI

[2/2] codegen  rewrite Tier 1, inject Tier 2 config, then compile and run the test suite
code_changes: SUCCESS
compile: SUCCESS
unit_testing: SUCCESS
  tests: total=2 passed=2 failed=0 skipped=0

Next
  open '~/work/spark-migrate-jobs/store-sales/code_changes/changes.html'
  spark-migrate-cli apply --job-id store-sales --branch spark-3.5-migration

• Analyze: Identifies compatibility issues and determines which changes can be automated.
• Codegen: Applies Tier 1 code rewrites and injects Tier 2 compatibility configurations.
• Compile: Builds the migrated application using Spark 3.5.5, Scala 2.12, and JDK 17.
• Unit testing: Runs the application's existing tests to verify the changes.

In this example, the migration completed in approximately 75 seconds, with all tests passing.

2. Review the changes

Open the generated changes.html report to review the modifications and their associated migration rules.

The following sections describe the key changes.

Build configuration

The tool updates the Scala and Spark versions and marks Spark dependencies as provided to use the Spark libraries available on the cluster.

-scalaVersion := "2.11.12"
+scalaVersion := "2.12.18"

-val sparkVersion = "2.4.8"
+val sparkVersion = "3.5.5"

-  "org.apache.spark" %% "spark-core" % sparkVersion,
+  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",

Application code

The tool replaces APIs that are no longer supported in Spark 3.x.

For example, the deprecated accumulator API is replaced with LongAccumulator:

-  def createInvalidRecordAccumulator(sc: SparkContext): Accumulator[Long] = {
-    sc.accumulator(0L, "invalidRecords")
+  def createInvalidRecordAccumulator(sc: SparkContext): org.apache.spark.util.LongAccumulator = {
+    sc.longAccumulator("invalidRecords")

Spark submission script

The tool updates the deprecated yarn-client master configuration while preserving client deployment mode.

-spark-submit \
-  --master yarn-client \
+spark-submit $SPARK_MIGRATE_CONF \
+  --master yarn --deploy-mode client \

Compatibility configuration

The tool generates a spark-migrate-injected.conf file containing the baseline compatibility settings and Parquet date configurations required by the application.

To view the explanation for a specific migration rule, run:

spark-migrate-cli explain SPARK-MIG-2001

3. Apply the changes

After reviewing the migration report, apply the changes to a new Git branch:

spark-migrate-cli apply --job-id store-sales --branch spark-3.5-migration

cd store-sales-etl
git status --short

 M build.sbt
 M project/build.properties
 M scripts/submit_nightly.sh
 M src/main/scala/com/acceldata/salesetl/Main.scala
 ...
?? spark-migrate-injected.conf

Next:

1. Build the application artifact using sbt assembly.
2. Commit the modified files, including spark-migrate-injected.conf.
3. Create a pull request for review.

4. Validate the migration

Before deploying the migrated application to production:

1. Run the original Spark 2.4.8 application and the migrated Spark 3.5.5 application using the same input data.
2. Compare their outputs to identify any differences in processing results.
3. Investigate and resolve any unexpected differences before cutover.

For detailed instructions, see Validate the data.

Result

The store-sales-etl application was successfully migrated from Spark 2.4.8 and Scala 2.11 to Spark 3.5.5 and Scala 2.12.

The migrated application compiled successfully, and both unit tests passed without requiring manual code changes. Production readiness remains subject to output validation and deployment checks.