Review, validate, and apply the migration

After migrating your Spark application, review the generated reports, validate the application output, and apply the changes to your repository.

1. Review the migration reports

Spark Migrate CLI generates two reports to help you review the migration:

Both reports can be opened in a browser and attached to a pull request for review.

To view the explanation for a specific migration rule, run:

spark-migrate-cli explain <rule_code>

spark-migrate-cli run -r <repository_path> --job-id <job_id> --exclude <rule_id_or_config_key>

2. Validate the data

Successful compilation and unit tests do not guarantee identical application output.

Before applying the migration, run the original Spark 2.4 application and the migrated Spark 3.5.5 application using the same input data. Compare their outputs to identify differences in schemas, row counts, aggregate values, and individual records.

To validate the Parquet output, run:

spark-migrate-cli validate --job-id store-sales \
  --baseline-data output-2.4/daily_summary \
  --candidate-data output-3.5/daily_summary \
  --key-columns storeId,saleDate

validate: dataset 'daily_summary' PASSED
  [PASS] schema: 4 column(s) match
  [PASS] row_count: 5 row(s) on both sides
  [PASS] aggregate: 2 numeric column(s) match within tolerance=0.0
  [PASS] keyed_row_diff: 0 mismatched row(s) across 2 column(s), key=storeId,saleDate

Example:

[FAIL] aggregate: 3 aggregate mismatch(es) (tolerance=0.0)
[FAIL] keyed_row_diff: 1 mismatched row(s) out of key=storeId,saleDate join (showing up to 100)

Example:

{
  "storeId": "S02",
  "saleDate": "2024-08-01",
  "total_amount": "baseline=2100.0 candidate=2000.0"
}

• Use --tolerance <value> to allow small floating-point differences.
• Use --ignore-validation to override failed validation. The override is recorded in the migration evidence.

3. Apply the migration

After reviewing the reports and validating the application output, apply the changes to a new Git branch:

spark-migrate-cli apply --job-id orders --branch spark-3.5-migration

Migrate directly from a Git repository

You can also migrate repositories hosted on GitHub, GitLab, or Bitbucket.

For example, to migrate a GitLab repository:

read -s SM_GIT_TOKEN && export SM_GIT_TOKEN

spark-migrate-cli migrate \
  --repo https://gitlab.example.com/data/payments-risk.git \
  --push

1. Clones the repository.
2. Analyzes and migrates the application.
3. Commits the changes to a new branch.
4. Pushes the branch to the remote repository.

You can then create a pull request for review.

4. Launch the application with compatibility configurations

Spark Migrate CLI automatically integrates the required configurations into the application's SparkSession.builder, spark-submit scripts, and test setup within the repository.

If the application is launched externally, such as through a scheduler or an Airflow DAG, include the generated configurations in the launch command.

For example:
spark-submit $(spark-migrate-cli conf --job-id orders) \
  --class com.example.Main job.jar

spark-migrate-cli conf --job-id orders --format defaults

After migrating the application to production, you can gradually remove legacy compatibility configurations. Validate the application after each change.

For details about each configuration, see code_changes/config_ledger.json.

5. Run the migrated application on Acceldata ODP

Spark Migrate CLI 1.1.0 targets Apache Spark 3.5.5. The migration rules and code changes are applicable to Acceldata ODP deployments running Spark 3.5.5.

However, ODP may bundle different dependency versions. Before running the migrated application on ODP, complete the following steps:

1. Update the Spark dependency version. Set the build configuration to the Spark version included in your ODP deployment, for example, 3.5.5.3.3.6.501-1002. Keep Spark dependencies marked as provided, and rebuild and test the application. Ensure your configured Maven or sbt repository can resolve the ODP artifacts.
2. Review bundled dependencies. Mark dependencies supplied by ODP as provided, where applicable. These may include Jackson, Guava, hadoop-aws, AWS SDK, Delta Lake, Iceberg, and Hudi.
3. Configure the Spark environment. Keep userClassPathFirst disabled and configure SPARK_HOME and application launch scripts to use the Spark 3 installation provided by ODP.

For additional configuration requirements and compatibility considerations, refer to the user guide included in the Spark Migrate CLI release package.