Migrate a PySpark application

This walkthrough demonstrates how to migrate a PySpark application from Spark 2.4.8 to Spark 3.5.5 using Spark Migrate CLI, including how to resolve compatibility issues that require manual intervention.

The example uses subscription-analytics, a nightly PySpark application that calculates subscription renewals and invoice schedules, groups customers by spending patterns using k-means clustering, and writes results to Parquet tables and a Delta ledger.

The application uses a spark-submit script with Delta Lake, spark-xml, and hadoop-aws dependencies specified through --packages. It also includes a pytest test suite.

Unlike migrations that can be completed automatically, this example contains two compatibility issues that require manual resolution. Spark Migrate CLI identifies these issues and stops the migration before applying changes, allowing you to resolve them before proceeding.