Reference and FAQ
This page provides a reference to Spark Migrate CLI commands, supported capabilities, limitations, and frequently asked questions.
Commands
The following table lists the available Spark Migrate CLI commands and their functions.
Command | Description |
run -r <repo> --job-id <id> | Analyzes, generates code changes, compiles, and tests the application. Stops when manual intervention is required. |
migrate --repo <git-url> [--push] | Migrates a remote Git repository, commits the changes to a new branch, and optionally pushes them. |
demo | Runs the built-in sample migration. |
doctor | Checks whether the required tools and dependencies are installed. |
analyze | Analyzes the application for compatibility issues. |
codegen | Generates code changes and compatibility configurations. |
compile | Compiles the migrated application. |
test | Runs the application's unit tests. |
validate | Compares the output data from the original and migrated applications. |
apply [--branch <name>] | Applies migration changes to the repository without committing them. |
conf | Displays the required Spark configurations for spark-submit or spark-defaults.conf. |
explain <code> | Displays detailed information about a migration rule. |
status | Displays the current migration job status. |
jobs | Lists migration jobs. |
report | Regenerates the migration report. |
bundle | Creates a ZIP archive of the migration job for sharing. |
abort | Discards the migration working copy. |
Common options
Option | Description |
--yes | Continues the migration past issues that require manual resolution. |
--exclude <rule_or_config_key> | Excludes a specific migration rule or configuration. |
--force | Reruns a migration step. |
--jobs-dir <dir> | Specifies a custom directory for migration jobs. |
To view all available options for a command, run:
spark-migrate-cli <command> --help
Exit codes
Exit code | Description |
0 | The command completed successfully. For run, this also includes stopping for manual review. |
1 | A migration step failed or the operation was refused. |
2 | The migrate command stopped for manual review without pushing changes. |
Each migration job generates a summary.json file containing the migration verdict, step results, and recommended next commands. You can use this file in automated scripts and CI/CD workflows.
Limitations
Spark Migrate CLI has the following limitations:
- Cluster execution: Does not run the application or make changes to a Spark cluster.
- Code generation: Uses predefined migration rules instead of AI-generated code changes.
- SQL migration: Analyzes .sql and .hql files but does not modify them. Dynamically generated SQL is identified as unchecked.
- Unsupported projects: Does not support notebooks, Gradle builds, Python 2-to-3 migration, or repositories containing both Java and Scala.
- Scala compatibility: Does not perform general Scala 2.11-to-2.12 source migrations.
- Spark functionality: Does not handle MLlib model behavior changes or Structured Streaming state migration.
- Logging configuration: Does not migrate Log4j 1 configurations to Log4j 2. Ignored log4j.properties files are reported.
- External launchers: Does not modify launch scripts outside the repository. Use the conf command to retrieve the required configurations.
- Pull requests: Does not automatically create pull requests.
Frequently asked questions
Does Spark Migrate CLI require cluster access or send my code to an external service?
No. Spark Migrate CLI runs locally and does not require access to a Spark cluster.
The only network requests are those made by your configured sbt, Maven, and coursier tools to download dependencies.
Does Spark Migrate CLI execute my application code?
The tool compiles the repository and runs its existing tests, similar to a standard build process.
Only analyze repositories that you trust and would normally build in your environment.
Do successful compilation and unit tests guarantee a successful migration?
No. Successful compilation and unit tests confirm that the migrated application builds and passes its tests on Spark 3.5.5.
They do not guarantee identical data-processing results. Use the validate command to compare outputs before deploying the migrated application to production.
Can I skip issues that require manual resolution?
Yes. Use the --yes option to continue the migration past mandatory manual-fix items.
However, the recommended approach is to resolve these issues before continuing. See the Migrate a PySpark application walkthrough for an example.
Why does the tool add spark.sql.legacy configurations?
Spark Migrate CLI adds a baseline set of spark.sql.legacy.* configurations to preserve Spark 2.4 behavior for specific operations.
It also adds configurations required by the application's code.
After migration, you can remove these configurations individually, validating the application output after each change.
How long does a migration take?
Migration time depends on the application's size, dependencies, and build and test duration.
The initial run may take several minutes because dependencies for Spark 2.4.8 and Spark 3.5.5 must be downloaded.
Subsequent migrations are generally faster. For example, the Scala walkthrough completed in approximately 75 seconds.
Documentation and support
The Spark Migrate CLI release package includes the following resources:
Resource | Description |
USER-GUIDE.md | Complete Spark Migrate CLI user guide. |
docs/rules/ | Migration rule catalog, including a page for each SPARK-MIG rule. |
examples/ | Sample applications, container configurations, and CI/CD pipeline templates. |
After installation, these resources are available in:
~/.spark-migrate/current/
When reporting an issue, generate a migration bundle using:
spark-migrate-cli bundle --job-id <id>
Have a suggestion?