Migrate Spark 2 Workloads to Spark 3

The Spark Migration Utility helps migrate existing Spark 2.x workloads running on Scala 2.11 to Spark 3.5.5 with Scala 2.12.

The utility provides a command-line interface (CLI) to automate parts of the migration process while allowing users to review and approve changes.

Prerequisites

• Download the Spark 2 to Spark 3 Migration Utility tarball and its SHA-256 checksum file from the Acceldata Repositories page.
• Ensure that you have access to the environment where you want to install the utility.

Install the Migration Utility

To install the Spark Migration Utility:

1. Download the migration utility tarball from the Acceldata Repositories page.
2. Verify the integrity of the downloaded tarball:

shasum -a 256 -c spark-migrate-cli-<version>.tar.gz.sha256

3. Extract the tarball:

tar -xzf spark-migrate-cli-<version>.tar.gz

4. Run the installation script:

spark-migrate-cli-<version>/install.sh --profile ~/.profile

5. The installer installs the utility under ~/.spark-migrate, including the Python virtual environment required by the migration engine.

Next Steps

After installing the utility, see the docs/USER-GUIDE.md file included in the extracted package for instructions on:

• Running the migration utility.
• Reviewing migration reports.
• Reviewing and approving proposed changes.
• Completing the migration process.