Installation, Configuration, and Job Submission

This page describes how to install and configure Apache SeaTunnel 2.3.13 in an ODP cluster using the Ambari Management Pack (MPack). It also explains how to prepare, configure, and submit data integration jobs using the supported SeaTunnel execution engines.

The page covers the following topics:

1. Install the MPack: Install the SeaTunnel MPack, add the SeaTunnel service through Ambari, and configure Kerberos authentication.
2. Key Use Cases: Understand common data integration patterns, including database ingestion, change data capture (CDC), lake format migration, multi-source ETL, cross-platform synchronization, and streaming pipelines.
3. Prepare and Submit a Job: Create HOCON configuration files, configure sources, transformations, and sinks, and submit jobs using the supported execution engines.
4. Engine-Specific Job Submission: Submit and manage jobs using Zeta, Flink, or Spark, including engine-specific configurations and prerequisites.
5. Engine Differences and Selection Guidelines: Compare execution engines and select an appropriate engine based on workload requirements, performance, streaming capabilities, and resource usage.
6. ODP Additions to Apache SeaTunnel 2.3.13: Review ODP-specific enhancements, including additional engine support, Kerberos integration, connector updates, and operational improvements.

Applies to: Apache SeaTunnel 2.3.13 MPack

Prerequisites:

• Ambari 2.7 or later
• ODP 3.3.6.6 or later, or ODP 3.4.3.0 or later

1. Install the MPack

This section describes how to install the SeaTunnel MPack on the Ambari server, add the SeaTunnel service to the cluster, and configure Kerberos authentication.

1.1 Install the MPack on the Ambari Server

Perform the following steps once on the Ambari server.

1. Stop the Ambari server.

sudo ambari-server stop

2. If an earlier version of the SeaTunnel MPack is installed, uninstall it.

sudo ambari-server uninstall-mpack --mpack-name=seatunnel-mpack

3. Install the SeaTunnel MPack.

sudo ambari-server install-mpack \
--mpack=/tmp/seatunnel-2.3.13.tar.gz --verbose

4. Restart the Ambari server.

sudo ambari-server restart

5. Verify the installation in the Ambari UI.
   a. Navigate to Admin > Stacks and Versions > ODP > 3.3 > Services.
   b. Verify that SeaTunnel version 2.3.13 is listed with the following components:
   • ZETA_MASTER
   • ZETA_WORKER
   • CLIENT

1.2 Add the SeaTunnel Service

1. In the Ambari UI, navigate to Actions > Add Service.
2. Select SeaTunnel.
3. On the Assign Masters page, assign SEATUNNEL_ZETA_MASTER to three hosts.
   Note: Use an odd number of master hosts to maintain a majority for Hazelcast partitions.
4. On the Assign Slaves and Clients page:
   • Assign SEATUNNEL_ZETA_WORKER to the same hosts or additional hosts.
   • Assign SEATUNNEL_CLIENT to each host from which users submit jobs.
5. On the Customize Services page, accept the default settings for a fresh installation or configure the following properties as required.
   | Configuration | Property | Description | |---|---|---| | hazelcast-master | master_rest_port | Specifies the master REST port. The default is 7654. Do not use the Ambari port 8080. | | seatunnel-env | jvm_tuning_profile | Specifies the JVM tuning profile. Supported values include default and azul_prime. | | plugin-config | content | Specifies the connectors installed through install-plugin.sh. If the MPack includes connector JAR files in the RPM, retain the default configuration. |
6. Review the configuration and select Deploy.
7. After installation, verify that the SeaTunnel services are active.

systemctl is-active seatunnel-master seatunnel-worker

8. Verify the Hazelcast cluster membership.

curl -s http://<master>:7654/hazelcast/rest/cluster | jq '.members | length'

1.3 Configure Kerberos

If Kerberos is enabled for the cluster, Ambari automatically generates the SeaTunnel principal and keytab during the Kerberize wizard.

The following table describes the generated Kerberos configuration.

Ambari populates the JAAS configuration with the required Kerberos values.

For validation instructions, see 06-security.md, section 1.3.

2. Key Use Cases

The SeaTunnel MPack supports six common integration patterns in the ODP data platform. Each pattern maps to a HOCON configuration template in docs/examples/, where available.

For multi-table enumeration, see 01-overview.md, section 2.7. For checkpointing and exactly-once processing, see section 2.6.

Engine Compatibility by Use Case

Important: SeaTunnel CDC connectors are not supported on Spark 3 or Spark 4 in batch or streaming mode. For more information, see section 5.

The engine recommendations are guidelines rather than mandatory requirements. You can submit the same HOCON configuration to a different compatible engine by changing the launcher without modifying the job configuration. For more information, see section 4.

3. Prepare and Submit a Job

3.1 Job Configuration Structure

Each SeaTunnel job uses a HOCON configuration file with three required top-level blocks: env, source, and sink. The transform block is optional.

The following example defines a batch job that reads data from MySQL, transforms it, and writes the results to HDFS.

env {
  # Engine-level settings
  parallelism = 4
  job.mode = "BATCH"           # or "STREAMING"
  checkpoint.interval = 10000  # STREAMING only
}

source {
  # One or more source definitions
  Jdbc {
    url = "jdbc:mysql://mysql-1:3306/salesdb"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "etl_ro"
    password = "..."
    query = "SELECT id, amount, ts FROM orders WHERE ts > '${last_ts}'"
    result_table_name = "orders_raw"
  }
}

transform {
  # Optional transformation definitions
  Sql {
    source_table_name = "orders_raw"
    result_table_name = "orders_typed"
    query = "SELECT id, CAST(amount AS DECIMAL(18,2)) AS amount, ts FROM orders_raw"
  }
}

sink {
  # One or more sink definitions
  HdfsFile {
    source_table_name = "orders_typed"
    fs.defaultFS = "hdfs://mycluster"
    path = "/warehouse/orders/dt=${today}"
    file_format_type = "parquet"
    kerberos_principal = "seatunnel/<host>@<REALM>"
    kerberos_keytab_path = "/etc/security/keytabs/seatunnel.service.keytab"
    hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml"
  }
}

This configuration supports:

• Fan-out: One source provides data to multiple sinks.
• Fan-in: Multiple sources provide data to a single sink.

3.2 Ready-to-Use Configuration Templates

The following templates are available in docs/examples/.

3.3 Submit a Job

Use the appropriate command for the target engine.

Zeta cluster mode (recommended default)

Submit the job to the running Zeta cluster without waiting for YARN resource allocation.

/usr/odp/current/seatunnel/bin/seatunnel.sh --config /path/to/job.conf

Run the job in a local JVM for development or smoke testing.

/usr/odp/current/seatunnel/bin/seatunnel.sh --config /path/to/job.conf -m local

/usr/odp/current/seatunnel/bin/start-seatunnel-flink-19-connector-v2.sh \
    -e run-application --target yarn-application \
    --config /path/to/job.conf

Spark on YARN

/usr/odp/current/seatunnel/bin/start-seatunnel-spark-3.5-connector-v2.sh \
    --master yarn --deploy-mode cluster --config /path/to/job.conf

4. Engine-Specific Job Submission

4.1 Zeta Engine

The Zeta engine uses bin/seatunnel.sh to submit jobs without requiring an external cluster manager or YARN resource allocation.

Cluster Mode

Submit the job to the running Zeta master.

sudo -u seatunnel bash -c '
  kinit -kt /etc/security/keytabs/seatunnel.service.keytab \
       seatunnel/$(hostname -f)@ADSRE.COM
  /usr/odp/current/seatunnel/bin/seatunnel.sh --config /path/to/job.conf
'

Local Mode

Run the job in a single JVM for development.

/usr/odp/current/seatunnel/bin/seatunnel.sh --config /path/to/job.conf -m local

Log Locations

Validated Test Results

The following results were recorded in the seatunnel-0 test environment running ODP 3.3.6.6-1004.

Start Time        : 2026-09-30 22:18:10
End Time          : 2026-09-30 22:18:12
Total Time(s)     : 2
Total Read Count  : 5
Total Write Count : 5
Total Failed Count: 0

4.2 Flink Engine (Flink 1.19 on ODP 3.3.6.6)

Flink jobs run on YARN in Application Mode.

ODP Hadoop includes commons-cli version 1.2, while Flink requires version 1.5 or later on the ApplicationMaster classpath. To avoid dependency conflicts, copy the Flink libraries to HDFS before submitting jobs.

Step 1: Copy the Flink Libraries to HDFS

Perform this step once for Flink 1.19.

# One-time seed of Flink 1.19 lib
FLINK_VER=1.19.1.3.3.6.6-1004 # rpm -q flink_<odp>-client

sudo -u hdfs bash -c "
  kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-<cluster>@<REALM>
  hdfs dfs -mkdir -p /flink/lib-${FLINK_VER}
  hdfs dfs -put -f /usr/odp/current/flink-client/lib/*.jar /flink/lib-${FLINK_VER}/
  hdfs dfs -chmod -R 755 /flink/lib-${FLINK_VER}
"

Step 2: Copy the SeaTunnel Runtime Libraries to HDFS

Perform this step once to make the SeaTunnel connectors and runtime libraries available.

sudo -u hdfs hdfs dfs -mkdir -p /seatunnel-runtime/2.3.13

sudo -u hdfs hdfs dfs -put -f /usr/odp/current/seatunnel/connectors/*.jar /seatunnel-runtime/2.3.13/

sudo -u hdfs hdfs dfs -put -f /usr/odp/current/seatunnel/lib/*.jar /seatunnel-runtime/2.3.13/

Step 3: Configure Flink in Ambari

1. Open the Ambari UI.
2. Navigate to FLINK > Configs > Advanced flink-conf.
3. Add the following properties.

yarn.provided.lib.dirs = hdfs:///flink/lib-1.19.1.3.3.6.6-1004
pipeline.classpaths = hdfs:///seatunnel-runtime/2.3.13/*

4. classloader.parent-first-patterns.additional = org.apache.seatunnel.api.

Step 4: Submit the Flink Job

sudo -u seatunnel bash -c '
  kinit -kt /etc/security/keytabs/seatunnel.service.keytab \
       seatunnel/$(hostname -f)@ADSRE.COM
  export FLINK_HOME=/usr/odp/current/flink-client
  export HADOOP_CONF_DIR=/etc/hadoop/conf
  export HADOOP_CLASSPATH=$(hadoop classpath)
  /usr/odp/current/seatunnel/bin/start-seatunnel-flink-19-connector-v2.sh \
      -e run-application --target yarn-application \
      --config /tmp/job.conf
'

Step 5: Verify the Job

1. List the YARN applications and verify that the SeaTunnel application is running.
2. 
   yarn application -list | grep SeaTunnel
3. 
   Review the application logs to verify job completion and record counts.
4. 
   yarn logs -applicationId <app-id> | grep -E "Job.*FINISHED|Total (Read|Write) Count"

Why is -m yarn-cluster not used?

The -m yarn-cluster option was deprecated in Flink 1.15. Use -e run-application --target yarn-application to submit jobs in YARN Application Mode.

Why must the Flink libraries be copied to HDFS?

ODP Hadoop includes commons-cli version 1.2 as a dependency. Flink 1.19 requires version 1.5 or later for CLI argument parsing.

Copying the Flink libraries to HDFS and configuring yarn.provided.lib.dirs ensures that the Flink version of the library is loaded first on the ApplicationMaster classpath without modifying Hadoop dependencies.

4.3 Spark Engine (Spark 3.5 or Spark 4.1)

Spark engine support depends on the ODP version and Java runtime.

Spark 4.1 requires JDK 17. ODP 3.4.3.0 uses JDK 17 by default and supports Spark 4.1 directly.

ODP 3.3.6.6 uses JDK 11 by default. Although Spark 4.1 starter and translation modules are included, you must use JDK 17 and enable the enable_spark4 MPack setting to run Spark 4.1 jobs.

Flink 2.2.1 integration is available only on ODP 3.4.3.0. ODP 3.3.6.6 includes Flink 2.0 placeholder components but does not include complete Flink 2.2.1 integration.

Engine Compatibility Matrix

Scala Compatibility

Following ODP-8153 and ODP-8153-336, SeaTunnel Spark starter and translation JAR files are independent of the Scala version.

The same binaries support:

• Spark 3.5 with Scala 2.12.18
• Spark 4.1 with Scala 2.13.17

Separate builds for each Scala version are not required. Spark provides the scala-library dependency at runtime, and SeaTunnel fat JAR files no longer include it.

For more information, see 11-odp343-stability-fixes.md, ODP-8153 Scala independence.

Submit a Spark 3.5 Job

Use the following command on either supported ODP version.

/usr/odp/current/seatunnel/bin/start-seatunnel-spark-3.5-connector-v2.sh \
    --master yarn --deploy-mode cluster --config /path/to/job.conf

Submit a Spark 4.1 Job

Ensure that JDK 17 is configured before submitting the job.

/usr/odp/current/seatunnel/bin/start-seatunnel-spark-4.1-connector-v2.sh \
    --master yarn --deploy-mode cluster --config /path/to/job.conf

Kerberos and Ranger Authentication

SeaTunnel Spark jobs use the standard Spark-on-YARN authentication workflow.

• To renew the submitting user's Kerberos ticket, specify the --principal and --keytab options.
• When HADOOP_USER_NAME is used, Ranger identifies the submitting user.

5. Engine Differences and Selection Guidelines

The following comparison is based on the official SeaTunnel engine comparison and laboratory observations.

5.1 Engine Selection Guidelines

Use Zeta for HDFS, Hive, and JDBC pipelines and interactive development. Zeta does not require YARN allocation, which reduces startup latency and avoids YARN queue resource consumption.

Use Flink for streaming CDC pipelines that require exactly-once processing, particularly when writing to Kafka or Iceberg. Flink is also suitable for workloads that exceed the memory capacity of a single Zeta worker JVM.

Use Spark for batch pipelines that depend on existing Spark connectors not provided by SeaTunnel, such as Delta writers or Snowflake native integrations. Spark is also suitable for workloads requiring broadcast joins with terabytes of reference data.

Important: Do not use Spark for SeaTunnel CDC pipelines. Use Zeta or Flink instead.

5.2 CDC Limitations on Spark

SeaTunnel CDC connectors are not supported on Spark 3 or Spark 4 in batch or streaming mode.

The limitation applies to the following connectors:

• connector-cdc-mysql
• connector-cdc-postgres
• connector-cdc-sqlserver
• connector-cdc-oracle
• connector-cdc-opengauss
• connector-cdc-mongodb
• connector-cdc-tidb

Upstream SeaTunnel explicitly disables Spark execution in CDC connector end-to-end tests.

@DisabledOnContainer(
        value = {},
        type = {EngineType.SPARK},
        disabledReason = "Currently SPARK do not support cdc")

Reason for the Limitation

SeaTunnel CDC sources generate coordinated streams that maintain checkpoint-aware state, including Debezium offsets and snapshot-to-binlog transition information.

This processing model is compatible with Flink DataStream and Zeta native streaming. However, the current SeaTunnel Spark translator does not support the same behavior through Spark Structured Streaming's micro-batch processing model.

Workaround: There is no Spark-specific workaround. Submit CDC pipelines using Zeta or Flink.

6. ODP Additions to Apache SeaTunnel 2.3.13

The ODP fork of Apache SeaTunnel 2.3.13 includes additional engine integrations, Kerberos enhancements, launcher script improvements, and dependency updates.

The source repository is github.com/acceldata-io/seatunnel.

For the complete commit-level differences and integration references, see 08-odp-additions.md.

6.1 ODP Enhancements