Performance Benchmarks

This section presents performance benchmarks for SeaTunnel execution engines on the ODP 3.3.6.6-1004 cluster. The benchmarks compare execution time, throughput, startup overhead, and connector performance across a common set of workloads.

Use these results to select an execution engine based on workload requirements. For engine selection guidance, see 02-usage.md, §3.3.

Benchmark Environment

The following configuration was captured in the lab on October 1, 2026.

Note: Spark 4.1.1 was excluded from this benchmark because it was not included in the bundled tarball for the tested ODP build. For more information, see 05-troubleshooting.md, §11.

1. Methodology

1.1 Workload Matrix

The benchmark uses the following workloads to evaluate execution engine overhead and connector performance.

Workloads W3–W7 cover JDBC-to-Hive, Kafka, MySQL CDC, Iceberg, and Hudi.

These workloads require connector-specific credentials and one-time topic or table configuration. The initial benchmark focused on comparing execution engine overhead under identical workload conditions. Connector-specific results are provided in §4.

1.2 Measurement Criteria

The benchmark uses the following measurement criteria.

Job execution time

Measure the total elapsed time from job submission to completion, including:

• JVM startup
• Connector class loading
• Source data processing
• Sink flushing
• Graceful shutdown
• YARN ApplicationMaster (AM) allocation, where applicable

Repeatability

Run each workload three times for each execution engine.

Report all three results and calculate the minimum, median, and maximum values.

For W2, clean the HDFS destination between runs.

Include the first run in the results to capture cold-start overhead.

Consistent configuration

Use the following settings across execution engines:

• Effective parallelism of 2
• Identical source and sink definitions
• Shared HOCON configuration files

Throughput

Calculate records per second by using:

The calculation includes startup overhead to reflect the end-to-end performance experienced by batch operators.

1.3 Shared HOCON Configurations

W1: Baseline configuration

The bench_w1.conf file defines a batch workload that generates one million records and sends them to the Console sink.

env {
  parallelism = 2
  job.mode = "BATCH"
}

source {
  FakeSource {
    plugin_output = "src"
    row.num = 1000000
    schema = {
      fields {
        id = bigint
        name = string
      }
    }
  }
}

sink {
  Console {
    plugin_input = "src"
    print.content = false
  }
}

The bench_w2.conf file extends the W1 configuration by adding a val = double field and replacing the Console sink with an HdfsFile sink.
sink {
  HdfsFile {
    plugin_input = "src"
    fs.defaultFS = "hdfs://ODP-Titan"
    path = "/tmp/seatunnel-bench-w2"
    file_format_type = "parquet"
    kerberos_principal = "seatunnel/_HOST@ADSRE.COM"
    kerberos_keytab_path = "/etc/security/keytabs/seatunnel.service.keytab"
  }
}

2. Captured Results

2.1 Performance Comparison

The following table summarizes the captured benchmark results.

Notes:

• W2 Flink completed successfully in all three runs after increasing TaskManager memory to 3 GB and correcting the YAML configuration. See §3.4.
• W6 generated one million rows using parallelism=2 and row.num=500000.
• W7 required parallelism=1 because of the Hudi multi-writer rollback issue. See §4.5.

W1 — End-to-end execution time (lower is better)

0 s15 s30 s45 s60 sZetaSpark 3.5Flink 1.19

Median or representative values reported in the source; Spark 4.1 was not tested.

2.2 Zeta Cluster — W1: FakeSource to Console

The W1 benchmark processed one million rows using the Zeta cluster.

Median execution time: 6 seconds.

Median throughput: Approximately 167,000 records/s.

The workload ran on a three-worker Zeta cluster with a parallelism setting of 2.

The following output shows the captured SeaTunnel job statistics:

Start Time       : 2026-10-01 03:24:47
End Time         : 2026-10-01 03:24:54
Total Time(s)    :                   7
Total Read Count : 1 000 000
Total Write Count: 1 000 000
Total Failed Count: 0

2.3 Zeta Cluster — W2: FakeSource to HDFS Parquet

The W2 benchmark processed 500,000 rows and wrote the output to HDFS in Parquet format.

Median execution time: 6 seconds.

Median throughput: Approximately 83,000 records/s.

The workload used three-way parallelism across HDFS DataNodes.

2.4 Spark 3.5 Local — W1: FakeSource to Console

The W1 benchmark processed one million rows using Spark 3.5 in local mode.

Median execution time: 12 seconds.

Approximate throughput: 87,000 records/s.

JVM and Spark driver startup accounted for approximately 5–6 seconds of execution time.

Although steady-state processing throughput was comparable to Zeta, Spark required approximately twice the end-to-end execution time because it initialized a new Spark context for each job.

For batch jobs running longer than 60 seconds, the relative impact of startup overhead decreases.

2.5 Spark 3.5 Local — W2: FakeSource to HDFS Parquet

The W2 benchmark processed 500,000 rows and wrote the output to HDFS in Parquet format.

Median execution time: 28 seconds.

Median throughput: Approximately 18,000 records/s.

Execution time was primarily affected by Spark context initialization and Parquet writer processing.

Zeta completed this workload approximately five times faster because its cluster was already running and its Parquet encoding path was more direct.

For larger workloads containing 10 million or more rows, Spark's executor parallelism is expected to reduce this performance difference.

3. Flink 1.19 on YARN — Fixes and Benchmark Results

The initial Flink 1.19 benchmark attempts revealed configuration and launcher issues.

Six issues were identified sequentially, with each subsequent issue becoming visible after resolving the previous one.

The fixes were incorporated into the SeaTunnel repository or Ambari mpack. The following sections describe the issues, resolutions, and benchmark results.

3.1 Issues Identified During Benchmark Execution

Issue 1: Flink configuration precedence

Flink 1.19 reads flink-conf.yaml when the file exists and falls back to config.yaml otherwise.

The lab contained a stale flink-conf.yaml with only the yarn.provided.lib.dirs property.

Because this file took precedence, Flink could not load the required memory configuration.

To resolve the issue, remove the stale file or add:

jobmanager.memory.process.size: 1024m
taskmanager.memory.process.size: 1536m

Issue 2: Incorrect deployment target

The launcher failed when using:

-t local

-t yarn-application

Issue 3: Missing HDFS staging directory

Flink requires the following HDFS staging directory:

/user/seatunnel/.flink/<app-id>

HdfsResource(..., action="create_on_execute")

Issue 4: Missing HBase delegation token classes

Flink's HBase delegation token provider was enabled by default, but the required HBase shaded JARs were unavailable in the Flink library directory.

The Flink mpack already disables the provider:

security.delegation.token.provider.hbase.enabled: false

Issue 5: HOCON configuration not available to YARN

The launcher did not make the HOCON configuration available in the ApplicationMaster working directory.

The AbstractFlinkStarter implementation now includes:

-Dyarn.ship-files="<config>"

Issue 6: Runtime archive path mismatch

The launcher passed the runtime archive as a relative path:

-Dyarn.ship-archives=runtime.tar.gz

However, the launcher checked for the archive in $APP_DIR, causing inconsistent archive creation and lookup.

For example, when launched from /tmp, the archive could be missing or empty because the required connector directories were not available in that location.

The launcher was updated to run:

cd "${APP_DIR}"

This ensures that:

• Connector, library, and plugin paths resolve correctly.
• runtime.tar.gz is created in $APP_DIR.
• Flink resolves the archive from the expected location.

Note: The same upstream issue exists in the Flink 1.13, 1.15, and 2.0 launchers. The ODP build includes only Flink 1.19, so the other launcher variants were not updated.

3.2 Supporting Mpack Configuration Fixes

The Flink launcher requires the following environment variables:

• FLINK_HOME
• FLINK_CONF_DIR

FLINK_HOME

Specifies the Flink installation directory.

The launcher invokes ${FLINK_HOME}/bin/flink. If the variable is not defined, shell execution with set -u fails.

FLINK_CONF_DIR

Specifies the directory containing the Flink configuration managed by Ambari.

Without this variable, the Flink CLI might read $FLINK_HOME/conf, which can differ from the managed configuration directory.

The mpack now exports these variables through seatunnel-env.xml when the corresponding directories exist.

if [ -d "/usr/odp/current/flink-client" ]; then
  export FLINK_HOME="${FLINK_HOME:-/usr/odp/current/flink-client}"
fi
if [ -d "/etc/flink/conf" ]; then
  export FLINK_CONF_DIR="${FLINK_CONF_DIR:-/etc/flink/conf}"
fi

3.3 Flink W1 — Captured Results

The W1 benchmark processed one million rows using Flink 1.19 in YARN application mode.

Median execution time: 57 seconds.

Median throughput: Approximately 17,500 records/s.

The approximate execution time breakdown was:

For this one-million-row workload, Flink required approximately:

• 10 times the end-to-end execution time of Zeta.
• 5 times the end-to-end execution time of Spark 3.5 local mode.

Flink is better suited to long-running, checkpointed streaming workloads where startup overhead becomes less significant over time.

Captured YARN applications:

application_1790815454946_0028
application_1790815454946_0029
application_1790815454946_0030

3.4 Flink W2 — Successful Results After Configuration Fixes

The W2 benchmark wrote one million rows to HDFS in Parquet format.

Median execution time: 61 seconds.

Median throughput: Approximately 16,400 records/s.

All three runs completed successfully after applying the following fixes.

Fix 1: Increase TaskManager memory

The initial TaskManager memory allocation of 1024 MB was insufficient for Parquet buffering and the Hadoop HDFS client.

The TaskManager encountered an OOM error during Parquet flushing. This caused a heartbeat timeout, after which YARN terminated the container.

Increase TaskManager memory to 3072 MB:

taskmanager:
  memory:
    process:
      size: 3072m
  numberOfTaskSlots: 2

Fix 2: Correct YAML parsing of JVM options

Adding 14 --add-opens options to env.java.opts.all increased the configuration value beyond approximately 1024 characters.

The SnakeYAML simple-key scanner rejected the value and generated an error similar to:

YamlEngineException: while scanning a simple key
could not find expected ':'

Enclose the JVM options in double quotation marks:

env:
  java:
    opts:
      all: "-XX:+UseG1GC --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=…"

Alternatively, distribute the options across:

• env.java.opts.jobmanager
• env.java.opts.taskmanager

This avoids maintaining a single long configuration value.

After applying both fixes, all three W2 runs completed successfully.

The median execution time of 61 seconds included approximately:

• 25 seconds for YARN startup.
• 30 seconds for Parquet writer initialization.
• 5 seconds for job execution.

3.5 Additional Issues Identified During Benchmarking

Two additional issues were identified while completing the Flink W1 benchmark.

Issue 7: Flink short-option binding

The FlinkCommandArgs.masterType property recognized only:

--master
--target

As a result, the YARN application branch in AbstractFlinkStarter.buildCommands() was not executed.

The fix added -t to the parameter declaration in FlinkCommandArgs.java.

Operators using --target were not affected.

Issue 8: JDK 17 module access restrictions

Flink 1.19 running on JDK 17 requires additional JVM module access options for YARN ApplicationMaster and TaskManager processes.

Without these options, the ApplicationMaster failed with:

java.lang.reflect.InaccessibleObjectException:
module java.base does not "opens java.util" to unnamed module

Add the required --add-opens options, including:

--add-opens=java.base/java.util=ALL-UNNAMED

For the full configuration, see 05-troubleshooting.md, §13.

4. Connector Workloads W3–W7 Using Zeta

This section presents connector-specific benchmark results captured using the Zeta cluster.

Flink and Spark comparisons were deferred. The results in this section focus on connector write performance rather than execution engine dispatch overhead.

4.1 W6 — FakeSource to Iceberg (HadoopCatalog)

Completed

The benchmark used bench_w6.conf to write data from FakeSource to Iceberg through HadoopCatalog with Kerberos authentication.

Iceberg achieved the highest connector-write throughput in this benchmark.

The connector buffers data in memory and performs an atomic snapshot commit instead of synchronizing each row individually.

The result also validated the Round 3 UGI proxy fix at scale.

4.2 W3 — MySQL to Hive EXTERNAL

Completed

The following configuration issues were identified and resolved during testing.

1. MySQL authentication compatibility

MySQL 8.x uses caching_sha2_password by default.

The connection failed with:

Public Key Retrieval is not allowed

ALTER USER ... IDENTIFIED WITH mysql_native_password BY ...;

The active Hive services were available only on specific hosts:

Connections to the other tested hosts were refused.

Configure the HOCON metastore_uri property to use the active Hive Metastore endpoint.

3. Hive connector classpath

The Hive connector uses provided-scope dependencies for hive-exec and hive-metastore.

Set the following classpath on all master and worker nodes:

SEATUNNEL_CLASSPATH=/usr/odp/current/hive-client/lib/*

Restart the SeaTunnel daemons to apply the updated classpath because the environment is loaded during JVM startup.

4. Rolling restart requirement

Do not restart all three Zeta masters simultaneously.

Doing so can trigger Hazelcast's cluster protection mechanism:

All node is lite, shutdown this cluster

Use the following restart sequence:

1. Restart one master.
2. Wait for Hazelcast cluster membership to converge.
3. Repeat for the remaining masters.
4. Restart the workers after the masters have rejoined.

4.3 W4 — FakeSource to Kafka JSON

Completed

The benchmark verified the Kafka offset using:

kafka-get-offsets.sh --topic bench_w4

bench_w4:0:200000

1. Duplicate JAAS configuration

An inline sasl.jaas.config property in HOCON conflicted with the JAAS configuration provided by the mpack:

-Djava.security.auth.login.config=/etc/seatunnel/conf/seatunnel-jaas.conf

Specifying another JAAS configuration caused:

LoginException: client is being asked for a password

The mpack-managed seatunnel-jaas.conf already includes the KafkaClient section with the service keytab and principal.

2. Kafka broker availability

Only seatunnel-1:6669 had an active Kafka broker during testing.

Configure bootstrap.servers to use the active broker.

For the lab configuration, use:

--replication-factor 1
--partitions 1

3. Initial SASL negotiation timeout

The first Kafka connection might encounter a SASL negotiation timeout.

Retrying the operation succeeds.

For more information, see 05-troubleshooting.md, §12.

4.4 W5 — MySQL CDC to Console Streaming

Completed on Zeta and Flink

The W5 benchmark evaluated a MySQL CDC initial snapshot containing 10,000 rows.

Zeta cluster results

The Console sink and two-second checkpoint I/O interval contributed to the execution time.

Flink 1.19 YARN results

The snapshot began at 06:48:17 and reached the first binlog event at 06:48:20.

Spark 3.5 results

Spark 3.5 did not support the tested CDC workload.

The execution failed with:

UnsupportedOperationException:
Flink ParallelSource don't support sending SourceEvent.
Please implement the SupportCoordinate marker interface
on the SeaTunnel source.

IncrementalSourceReader.pollNext
ParallelSource.run
ParallelBatchPartitionReader

SeaTunnel CDC sources require coordination between split enumerators and readers for:

• Watermark alignment
• Split assignment
• Binlog position management

The Zeta engine supports this coordination directly.

The Flink translation layer implements it through Flink OperatorEvent.

The Spark translation layer, ParallelBatchPartitionReader, does not provide the required coordination mechanism.

This limitation affects the following CDC sources when using Spark 3.5 or 4.1:

• MySQL CDC
• PostgreSQL CDC
• SQL Server CDC
• Oracle CDC

The failure is an architectural limitation rather than a connector defect.

CDC performance comparison

Flink's snapshot throughput was approximately 20 times higher than Zeta's after startup.

However, the end-to-end execution times were comparable for the 10,000-row snapshot because Flink required approximately 60 seconds to initialize the YARN application.

For sustained streaming workloads, Flink is expected to provide better throughput as startup overhead becomes less significant.

For short, one-time CDC snapshots containing fewer than one million rows, Zeta avoids YARN startup overhead.

Spark is not suitable for the tested CDC workloads.

MySQL JDBC URL configuration

MySQL 8.x uses caching_sha2_password authentication by default.

For the tested configuration, use the following JDBC URL:

url = "jdbc:mysql://host:3306/db?useSSL=false&allowPublicKeyRetrieval=true"

Public Key Retrieval is not allowed

To recover a blocked host, run:

FLUSH HOSTS;

The MySQL CDC snapshot provides a bounded phase with a defined completion event, making it suitable for streaming benchmarks.

For more information about why FakeSource streaming cannot be used for this comparison, see §4.6.

4.5 W7 — FakeSource to Hudi Copy on Write

Completed with parallelism set to 1

Root cause of multi-writer failures

The SeaTunnel Hudi connector encountered a race condition when multiple writers attempted to write to the same table.

With parallelism greater than 1:

1. Multiple writers independently invoked BaseHoodieWriteClient.startCommit().
2. Each writer invoked rollbackFailedWrites().
3. One writer attempted to remove another writer's in-flight commit marker.
4. Hudi failed to delete the in-flight commit.
5. The failure was wrapped by HudiRecordWriter.doAsLoginUser as a COMMON-14 Kerberos authorization error.

The underlying exception was:

HoodieIOException: Could not delete instant ... INFLIGHT

Required configuration

Set the Hudi sink parallelism to 1:

env {
  parallelism = 1
  job.mode = "BATCH"
}

An upstream fix would require either:

• Commit serialization across parallel Hudi writers through a coordinator.
• A connector-level maximum parallelism setting in SeaTunnel.

Until such a fix is available, use parallelism=1 for Hudi sinks in this tested configuration.

4.6 Recommended Streaming Benchmark Pattern

The initial streaming benchmark used FakeSource in STREAMING mode.

However, FakeSource did not terminate automatically after generating the number of records specified by row.num.

This is the behavior observed with upstream SeaTunnel 2.3.13.

The source reader remained active indefinitely, and all attempts reached the 90-second timeout without producing conclusive throughput measurements.

Recommended approach

Use MySQL CDC (W5) for streaming benchmarks.

The initial CDC snapshot provides a bounded phase with a defined completion event.

To compare execution engines:

1. Use the same bench_w5_stream.conf configuration.
2. Configure the same env.checkpoint.interval value.
3. Measure the snapshot execution time until Total Write Count matches the source row count.
4. Calculate the average records processed per second during the snapshot.
5. Optionally, measure the 95th-percentile (p95) end-to-end latency for events inserted during the subsequent binlog streaming phase.

For Flink, run the same configuration through the Flink launcher.

The source notes that further streaming comparisons were deferred to a dedicated session. The Flink CDC results captured in §4.4 provide the available initial comparison.

5. Observations from the Captured Results

The following observations summarize execution engine and connector performance in the tested environment.

Zeta performance for short-running workloads

Zeta provided the lowest end-to-end execution time for workloads completing in fewer than 30 seconds.

Its long-running cluster processes and Hazelcast-based coordination eliminated most per-job startup overhead.

The one-million-row Console benchmark completed in six seconds, including client startup.

Spark 3.5 startup overhead

Spark 3.5 local mode required approximately five seconds to initialize the driver JVM before processing records.

For W1, startup accounted for approximately 45% of the execution time.

For W2, Parquet writer processing was the primary bottleneck.

Spark achieved approximately 18,000 records/s because the writer was not parallelized in local mode.

Zeta HDFS Parquet throughput

Zeta achieved approximately 83,000 records/s for the HDFS Parquet workload.

The primary bottlenecks were HDFS replication and Parquet encoding rather than Zeta job dispatch.

Increasing parallelism to 12, across three workers with four tasks each, was estimated to improve throughput by approximately two to three times for wider records.

This estimate was not validated in the captured runs.

Flink YARN startup overhead

Flink required approximately 25 seconds for YARN ApplicationMaster allocation and initial container startup.

For workloads completing in fewer than 60 seconds, Zeta provided better end-to-end performance.

Flink is more suitable for long-running streaming workloads where startup overhead is distributed across hours of processing.

Iceberg connector performance

Iceberg achieved approximately 333,000 records/s, the highest connector-write throughput observed in this benchmark.

The connector buffers records in memory and commits an atomic snapshot, avoiding per-record synchronization.

By comparison, HDFS Parquet writes were constrained by replication and filesystem synchronization.

Sustained streaming performance

Flink is expected to outperform Zeta for sustained, checkpointed streaming workloads.

However, the benchmark did not capture a complete, long-duration streaming comparison because FakeSource in streaming mode did not terminate automatically.

The MySQL CDC results in §4.4 provide an initial snapshot performance comparison.

6. Reproduce the Benchmarks

The benchmark configuration templates are available at:

docs/examples/bench_w1.conf
docs/examples/bench_w2.conf

Zeta Cluster

Run the W1 benchmark using the Zeta cluster:

sudo -u seatunnel bash -c '
  source /etc/seatunnel/conf/seatunnel-env.sh
  /usr/odp/current/seatunnel/bin/seatunnel.sh --config /tmp/bench_w1.conf
'

Spark 3.5 Local

Run the W1 benchmark using Spark 3.5 in local mode:

sudo -u seatunnel bash -c '
  source /etc/seatunnel/conf/seatunnel-env.sh
  export SPARK_HOME=/usr/odp/current/spark3-client
  export HADOOP_CONF_DIR=/etc/hadoop/conf
  /usr/odp/current/seatunnel/bin/start-seatunnel-spark-3.5-connector-v2.sh \
      --master local --deploy-mode client --config /tmp/bench_w1.conf
'

Flink 1.19 on YARN

After applying the configuration changes described in §3.2, run the W1 benchmark using Flink 1.19 in YARN application mode:

sudo -u seatunnel bash -c '
  export FLINK_HOME=/usr/odp/current/flink-client
  export FLINK_CONF_DIR=/etc/flink/3.3.6.6-1004/0
  export HADOOP_CLASSPATH=$(hadoop classpath)
  kinit -kt /etc/security/keytabs/seatunnel.service.keytab seatunnel/$(hostname -f)@ADSRE.COM
  /usr/odp/current/seatunnel/bin/start-seatunnel-flink-19-connector-v2.sh \
      -e run-application -t yarn-application --config /tmp/bench_w1.conf
'