Troubleshooting and Tuning

This section describes common issues, their causes, and recommended resolutions when running Apache SeaTunnel on ODP. It also provides configuration recommendations for improving performance and reliability.

Issues that have already been resolved in the ODP Management Pack (MPack), connectors, launcher scripts, and Flink patches are excluded from the active troubleshooting procedures.

For additional information, see:

• SeaTunnel upstream FAQ: Engine-related failures, including Flink ApplicationMaster out-of-memory (OOM) errors, Spark executor OOM errors, and YARN containers exceeding physical memory limits.
• 02-usage.md, section 3.2.2, Flink engine: Initial HDFS library setup and yarn.provided.lib.dirs configuration.

1. Job finishes successfully, but no data is written to the sink

Symptom

A SeaTunnel job reports a FINISHED status with no failed records, but the destination remains empty.

Example output:

Total Read Count   : 10
Total Write Count  : 10
Total Failed Count : 0
Job status         : FINISHED

Cause

The worker JVM might suppress sink-side errors, particularly authorization failures.

The write counter tracks records sent to the writer buffer, not records acknowledged by the destination.

Resolution

1. Check the SeaTunnel worker log for errors associated with the job ID:

JOB_ID=<job-id>
grep -iE "$JOB_ID|kafka.*[Ee]xception|AccessControl|AuthorizationException|Cluster authorization failed" \
/var/log/seatunnel/seatunnel-engine-worker.log

2. Resolve the authorization issue based on the sink type:
   • Kafka: Grant the required access control list (ACL) permissions to the seatunnel principal. See 03-connectors.md, Fake → Kafka.
   • HDFS: Update the Ranger HDFS policy or configure ownership and ACL permissions using hdfs dfs -chown or -setfacl.
   • HBase: Grant the required permissions using the HBase shell:

grant 'seatunnel', 'RWXCA', '@<namespace>'

2. Large connector JAR files increase memory usage

Symptom

The Hudi and Iceberg connector JAR files contain a large number of shaded dependencies.

The following table summarizes the JAR sizes and included classes identified during testing.

Cause

The connector JAR files include redundant dependencies, increasing memory usage and plugin download time.

This is not a runtime defect in the tested cluster because shading relocates the connector dependencies into private namespaces, preventing conflicts with ODP Hadoop and ZooKeeper libraries.

Recommendation

Consider excluding the following dependencies from connector-hudi/pom.xml and marking them as provided:

org.apache.hadoop:*
org.apache.zookeeper:*

These changes are optional optimizations and are not required to resolve a runtime issue.

3. Zeta cluster members fail to join after a restart

Symptom

After restarting the Zeta master services, the systemd units report an active status, but the /hazelcast/rest/cluster endpoint displays fewer cluster members than expected.

The logs contain messages indicating that LiteNodeDropOutTcpIpJoiner has blacklisted a peer.

Cause

Multiple master nodes restart simultaneously. An initial connection failure causes the joiner to blacklist the affected peer for 60 seconds.

Resolution

1. Restart the Zeta master nodes sequentially, allowing at least five seconds between restarts.
2. Use the rolling restart script provided with the MPack:

docs/examples/rolling_restart.sh

3. Alternatively, use Restart All in the Ambari UI, which restarts the services sequentially.
4. Verify that the expected cluster members appear in the /hazelcast/rest/cluster endpoint.

4. Remove obsolete SeaTunnel log rotation configuration

Symptom

An existing installation contains the following configuration file:

/etc/logrotate.d/seatunnel

Older SeaTunnel builds included a separate logrotate configuration. Current MPack builds no longer require this file because log rotation is managed by the respective logging mechanisms.

• .out files are obsolete because systemd captures standard output through journald.
• .log files are rotated by Log4j2 RollingFileAppender daily or when they reach 100 MB, with a retention count of 30.
• -gc.log* files use JVM garbage collection log rotation with filecount=5.

Using logrotate with copytruncate alongside Log4j2 rotation can cause conflicts and corrupt log rotation boundaries.

Resolution

1. Remove the obsolete logrotate configuration:

sudo rm -f /etc/logrotate.d/seatunnel

2. Remove obsolete .out files from earlier installations:

sudo rm -f /var/log/seatunnel/*.out

5. Hive sink job finishes successfully, but queries return no rows

Symptom

A SeaTunnel Hive sink job reports the following results:

• Total Write Count: 80
• Job status: FINISHED
• Failed records: 0

However, running SELECT COUNT(*) returns zero rows, even though ORC files exist in the HDFS table location.

Cause

ODP Hive 4.1 configures managed tables to use the ACID transactional format by default.

ACID tables require transaction-managed base and delta file layouts. The SeaTunnel Hive sink writes standard ORC files that are not recognized as transactional records by managed ACID tables.

Resolution

Create an external Hive table for the SeaTunnel sink:

CREATE EXTERNAL TABLE default.st_hive_ext (
 id INT, name STRING, ts STRING
) STORED AS ORC
LOCATION 'hdfs://ODP-Titan/tmp/seatunnel-smoke/hive_ext';

TBLPROPERTIES ('transactional'='false')

6. MySQL CDC source does not support batch jobs

Symptom

A MySQL CDC job fails with the following error:

‘MySQL-CDC’ source don’t support off-line job

The MySQL CDC connector supports streaming jobs only.

Resolution

Configure the job to use STREAMING mode:

env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 2000
}

7. JDBC driver is not found on a remote worker

Symptom

A SeaTunnel job fails with the following error:

The jar file file:/usr/odp/current/seatunnel/plugins/jdbc/lib/mysql-connector-java.jar
can not be found in node seatunnel-1.<domain>, please ensure that the deployment
paths of SeaTunnel on different nodes are consistent.

SeaTunnel distributes the JDBC driver file path from the submission host to remote workers.

If the driver filename or path differs across nodes, the remote workers cannot locate the required JAR file.

Resolution

1. Ensure that the JDBC driver has the same filename and directory path on every SeaTunnel node.
2. Navigate to the JDBC plugin directory:

cd /usr/odp/current/seatunnel/plugins/jdbc/lib/

3. Remove alternate connector filenames and symbolic links:

find . -type f -name 'mysql-connector-*' ! -name 'mysql-connector-j-8.0.32.jar' -delete
find . -type l -delete

4. Restart the SeaTunnel workers on all hosts.

8. Ranger repository creation fails because required users do not exist

Symptom

After installing Ranger, HDFS, YARN, Hive, or HBase components fail to start with errors similar to the following:

{"statusCode":400,"msgDesc":"Operation denied. User name: rangerlookup specified in policy does not exist in ranger admin."}
{"statusCode":400,"msgDesc":"Operation denied. User name: ambari-qa specified in policy does not exist in ranger admin."}

The issue occurs because of two upstream defects:

1. The Ranger Admin database initialization does not create the rangerlookup user, although the default plugin policy templates reference it.
2. The Ambari create_ranger_repository() function in ranger_functions_v2.py does not verify that users referenced in the default policies exist before submitting the repository creation request.

The YARN policy also explicitly references the ambari-qa user.

Resolution

1. Configure the Ranger Admin endpoint and credentials:

RANGER=http://<ranger-admin>:6080
AUTH="admin:<ranger-admin-pwd>"

2. Create the required users through the Ranger REST API:

for u in rangerlookup ambari-qa seatunnel hdfs yarn hive hbase spark flink; do
 curl -s -u "$AUTH" -H "Content-Type: application/json" -X POST \
 "$RANGER/service/xusers/secure/users" \
 -d "{\"name\":\"$u\",\"password\":\"User@2026\",\"firstName\":\"$u\",\"status\":1,\"isVisible\":1,\"userSource\":0,\"userRoleList\":[\"ROLE_USER\"]}"

3. done

4. Assign the required administrative role to rangerlookup:

curl -s -u "$AUTH" -H "Content-Type: application/json" -X PUT \
 "$RANGER/service/xusers/users/rangerlookup" \
 -d '{"name":"rangerlookup","userRoleList":["ROLE_SYS_ADMIN"]}'

5. Remove the stale Ranger policy cache directories on each affected host:

sudo rm -rf /etc/ranger/<CLUSTER>_hadoop /etc/ranger/<CLUSTER>_yarn

6. Restart the affected master components through the Ambari API.

Verification

Run the following command to list the Ranger services:

curl -s -u "$AUTH" "$RANGER/service/public/v2/api/service" | jq '.[] | .name'

ODP_<CLUSTER>_hadoop
ODP_<CLUSTER>_yarn
ODP_<CLUSTER>_hive
ODP_<CLUSTER>_hbase
ODP_<CLUSTER>_kafka3
ODP_<CLUSTER>_tag

Security note: The example uses a fixed password from the source procedure. Replace it with a secure, unique password before running the command in an operational environment.

9. Ranger policy creation fails because the Hadoop group does not exist

Symptom

A Ranger policy creation request fails with the following error:

Group name: hadoop does not exist in ranger admin

Ranger UserSync has not synchronized the required operating system groups.

Groups referenced in Ranger policies must exist in Ranger before the policies can be created.

Resolution

Use one of the following approaches:

• Remove group references from the policy request and configure permissions using existing users.
• Wait for Ranger UserSync to synchronize the required groups, or trigger synchronization before creating the policy.

10. Kafka clients time out when connecting to a Kerberos-enabled cluster

Symptom

A Kafka client connects to the cluster but fails with the following error:

TimeoutException: Timed out waiting for a node assignment. Call: listTopics

ODP Kafka3 uses port 6669 for SASL connections on Kerberos-enabled clusters.

The default plaintext port 6667 is not available in this configuration.

Resolution

1. Configure the Kafka sink to use port 6669:

sink {
  Kafka {
    bootstrap.servers = "seatunnel-1.<domain>:6669"
    ...
  }

2. }

3. In Ambari, navigate to Kafka3 > Configs > Advanced kafka-broker.
4. Verify the listeners configuration.

11. Kafka administrative commands fail with authorization errors

Symptom

Running Kafka administrative commands as the kafka operating system user fails with an authorization error.

Example:

ClusterAuthorizationException: Request ... is not authorized.

The KAFKA_OPTS environment variable references a JAAS configuration that authenticates using a principal without Kafka superuser permissions.

Kafka administrative tools use the authenticated JAAS principal, not the operating system user executing the command.

Resolution

1. Create a dedicated Kafka administrator JAAS configuration:

cat > /tmp/kafka_admin_jaas.conf <<EOF
KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true storeKey=true
  keyTab="/etc/security/keytabs/kafka.service.keytab"
  principal="kafka/<fqdn>@<REALM>"
  serviceName="kafka";
};
EOF

2. Set the KAFKA_OPTS environment variable:

export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/kafka_admin_jaas.conf"

3. Create the /tmp/kafka_admin.properties file with the following configuration:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

4. Include the following option when running kafka-topics.sh or kafka-acls.sh:

--command-config /tmp/kafka_admin.properties

12. Kafka SASL authentication intermittently times out

Symptom

Kafka administrative commands or SeaTunnel Kafka clients intermittently fail with the following messages:

ERROR TimeoutException: Timed out waiting for a node assignment. Call: createTopics
WARN [Principal=kafka/...@...]: TGT renewal thread has been interrupted and will exit.

Cause

A timing conflict can occur between Kerberos ticket-granting ticket (TGT) renewal and the initial SASL authentication handshake.

If the cached ticket expires, the Kafka TGT renewal thread might be interrupted during authentication. As a result, the client cannot establish a broker connection before the request times out.

Resolution

Use one of the following approaches:

1. Retry the failed command.
2. Increase request.timeout.ms and metadata.max.age.ms in the Kafka administrator properties for automated or scripted operations.
3. Obtain a fresh Kerberos ticket before running Kafka commands:

kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/$(hostname -f)

13. Tune SeaTunnel performance

This section describes configuration recommendations for Zeta heap memory, parallelism, checkpoints, Kafka producers, and JDBC sources.

13.1 Configure Zeta master heap memory

The default Zeta master heap settings in seatunnel-env are:

-Xms1g -Xmx1g

1. In Ambari, navigate to SEATUNNEL > Configs > Advanced seatunnel-env.
2. Set the following property:

seatunnel_master_heap = 4096

13.2 Configure Zeta worker heap memory and parallelism

Configure the worker heap to accommodate the expected workload.

Use the following sizing guideline:

Worker heap = parallelism × batch_size × row_size × 3

13.3 Configure checkpoint intervals

Configure the checkpoint interval based on the workload type.

13.4 Tune Kafka producer batch settings

Add the following properties to sink.Kafka.kafka.config:

batch.size       = "524288"
linger.ms        = "20"
compression.type = "snappy"
buffer.memory    = "268435456"
retries          = "10"

The retry configuration helps Kafka producers recover from temporary connection interruptions during Kerberos ticket renewal.

13.5 Configure JDBC source parallelism

To enable parallel JDBC reads, configure both partition_column and partition_num.

Set partition_num to match env.parallelism.

Without these settings, the JDBC source runs in a single thread, regardless of the configured parallelism.

Select a monotonically increasing primary key or an appropriate hash-based partitioning column.

14. Spark 4.1 starter JAR is missing from the SeaTunnel installation

Resolved in ODP-8153 and ODP-8153-336

Symptom

Running start-seatunnel-spark-4.1-connector-v2.sh fails with the following error:

ClassNotFoundException: org.apache.seatunnel.core.starter.spark.SparkStarter

Earlier SeaTunnel builds for ODP 3.3.6.6 included only the Flink 1.19 and Spark 3.5 starter JAR files.

The Spark 4.1 starter JAR was excluded because its module required Java 17, while ODP 3.3.6.6 used Java 11 by default.

Resolution status

The issue was resolved through the following changes:

• Added the seatunnel-spark-4.1-starter and seatunnel-translation-spark-4.1 modules to both ODP release lines.
• Updated the starter JAR files to support Scala 2.12 and Scala 2.13 through the ODP-8153 Scala-independent refactoring.
• Added automatic SPARK_HOME and SPARK_MAJOR_VERSION exports to the MPack's seatunnel-env.sh configuration.

The runtime requirements differ by ODP version.

Troubleshooting

If the issue persists, verify that:

1. The installed SeaTunnel RPM includes the ODP-8153 fix.
2. The enable_spark4 property is enabled.
3. The deployed Java version is Java 17.

As a temporary alternative, use Spark 3.5. The source testing indicated approximately 5–10% lower throughput compared with Spark 4.1 for the tested workloads.

15. Flink 1.19 and 2.2.1 fail on Java 17 because JVM modules are inaccessible

Symptom

A Flink YARN ApplicationMaster fails during job execution with the following error:

java.lang.reflect.InaccessibleObjectException:
module java.base does not "opens java.util" to unnamed module

Flink 1.19 and 2.2.1 use reflection to access internal Java modules.

Java 17 restricts reflective access to these modules unless the required --add-opens JVM options are configured.

Although the Flink CLI includes these options for the client JVM, the YARN ApplicationMaster and TaskManager use the options defined in env.java.opts.all.

The tested ODP Flink MPack configuration did not include the required options.

Resolution

1. Open the Flink config.yaml file.
2. Add the required JVM options to env.java.opts.all, preserving existing options:

env:
  java:
    opts:
  all: "-XX:+UseG1GC --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.net.util=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED"

3. Ensure that all JVM options are specified on a single line in the YAML configuration.

16. SeaTunnel CDC sources are not supported on Spark

Symptom

SeaTunnel change data capture (CDC) jobs fail when executed using Spark 3.5 or Spark 4.1.

On Spark 3.5, the job fails with the following error:

java.lang.UnsupportedOperationException: Flink ParallelSource don't support sending SourceEvent.
Please implement the SupportCoordinate marker interface on the SeaTunnel source.
    at …IncrementalSourceReader.pollNext(…)
    at …ParallelSource.run(…)
    at …spark.source.partition.batch.ParallelBatchPartitionReader.lambda$prepare$0(…)

Caused by: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN;
  Could not deserialize ATN with version 3 (expected 4).
    at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:56)
    at io.debezium.ddl.parser.mysql.generated.MySqlLexer.<clinit>(MySqlLexer.java:6278)
    at …MySqlIncrementalSource.tableChanges(MySqlIncrementalSource.java:142)

SeaTunnel CDC sources use the SupportCoordinate marker interface to coordinate source events, including:

• Split assignment to readers
• Binlog-position watermarks
• Snapshot-to-binlog phase transitions

The Zeta engine and Flink translation layer support source event delivery. The Spark translation layer does not.

Spark 4.1 also has an ANTLR dependency incompatibility. Spark 4.1 provides ANTLR 4.9, while the Debezium-generated MySQL lexer requires ANTLR 4.13 or later.

Even if the ANTLR incompatibility is resolved, the Spark translation layer does not support the required source event coordination.

Resolution

Run CDC jobs using the Zeta or Flink engine.

For one-time historical data ingestion from MySQL, PostgreSQL, or Oracle, use the standard JDBC source with Spark instead of a CDC source.

For benchmark results and test findings, see:

• 07-benchmarks.md, section 4.4, W5 three-engine streaming comparison
• 10-odp343-lab-findings.md, section 7, W5

17. Flink 2.2.1 does not support the run-application command

Symptom

Submitting a Flink 2.2.1 job using start-seatunnel-flink-2-connector-v2.sh fails with one of the following errors:

java.lang.IllegalArgumentException: The given host:port ('yarn-per-job') doesn't contain a valid port
    at org.apache.flink.util.NetUtils.validateHostPortString(NetUtils.java:120)
    at org.apache.flink.client.cli.DefaultCLI.toConfiguration(DefaultCLI.java:64)

java.lang.IllegalStateException: No ExecutorFactory found to execute the application.
    at org.apache.flink.core.execution.DefaultExecutorServiceLoader.getExecutorFactory(…)

Flink 2.2.1 removed the flink run-application subcommand and the yarn-per-job execution target.

Flink 2.2.1 uses the following execution targets:

• flink run --target yarn-application
• flink run --target yarn-session

The SeaTunnel AbstractFlinkStarter implementation for Flink 1.x generates the obsolete flink run-application command, which Flink 2.2.1 rejects.

Resolution

Update AbstractFlinkStarter.buildCommand() to generate flink run when the starter engine is FLINK2.

When submitting the job, use:

-t yarn-application

-m yarn-per-job

Until the starter patch is available, submit the job directly through the Flink CLI:

${FLINK_HOME}/bin/flink run --target yarn-application -c org.apache.seatunnel.core.starter.flink.SeaTunnelFlink <starter-jar> --config <conf> --name <name>

For implementation details, see 10-odp343-lab-findings.md, F-11.

18. Flink 2.2.1 TaskManager fails with a Netty LinkageError

Symptom

The Flink TaskManager container fails to register with the cluster.

The aggregated YARN logs contain one of the following errors:

IncompatibleClassChangeError: Class org.apache.flink.shaded.netty4.io.netty.handler.flush.FlushConsolidationHandler
  does not implement the requested interface
  org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandler

LinkageError: loader constraint violation: … Bootstrap/AddressResolverGroup —
  SubmoduleClassLoader (parent='platform') has Bootstrap
  'app' loader has AddressResolverGroup

The flink-rpc-akka.jar module is embedded in flink-dist-2.2.1.jar and contains its own shaded Netty classes.

The ComponentClassLoader.loadClass() implementation evaluates the component-first package prefix before the owner-first prefix.

Because the broader org.apache.flink prefix matches first, the more specific org.apache.flink.shaded. prefix is not applied.

Consequently, shaded Netty classes are loaded by both SubmoduleClassLoader and the application class loader, causing class linkage failures.

Resolution

Apply the two-file Flink source patch described in 10-odp343-lab-findings.md, F-18.

The patch includes the following changes:

1. Update SubmoduleClassLoader.java to add org.apache.flink.shaded. to OWNER_FIRST_PACKAGES.
2. Update ComponentClassLoader.java to use the longest matching package prefix when resolving component-first and owner-first loading rules.

After applying the patch, rebuild the Flink components:

cd /path/to/flink
mvn -pl flink-core,flink-dist -am -Dmaven.test.skip=true -Dfast clean package

flink-dist/target/flink-dist_2.12-2.2.1.3.4.3.0-1013.jar

1. Move the existing JAR file to a backup location outside the Flink lib directory:

sudo mv /usr/odp/current/flink-client/lib/flink-dist-2.2.1.3.4.3.0-1013.jar /root/

2. Install the updated JAR file:

sudo install -o flink -g flink -m 644 \
 /path/to/new/flink-dist_2.12-2.2.1.3.4.3.0-1013.jar \
/usr/odp/current/flink-client/lib/flink-dist-2.2.1.3.4.3.0-1013.jar

Additional configuration

For SeaTunnel Flink 2 YARN jobs, consider the following settings:

• Increase JobManager and TaskManager process memory for large workloads, such as one-million-row Fake sources:

-Djobmanager.memory.process.size=1536m
-Dtaskmanager.memory.process.size=1536m

• Configure the checkpoint directory for streaming jobs:

-Dstate.checkpoints.dir=hdfs:///user/seatunnel/flink-checkpoints

• The default /apps/odp/flink directory is owned by flink:hadoop with 755 permissions, preventing the seatunnel user from writing checkpoints.

19. SeaTunnel Spark 4.1 starter uses the Spark 3 submission environment

Symptom

A SeaTunnel Spark 4.1 job fails with Scala binary compatibility errors, such as IncompatibleClassChangeError or missing WrappedArray symbols.

The spark-submit output displays the following message:

Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set —
the lowest available Spark version will be used by default

If SPARK_HOME is not configured, the start-seatunnel-spark-4.1-connector-v2.sh script uses the system spark-submit command.

When Spark 3 and Spark 4 are installed, the submission wrapper selects Spark 3 unless SPARK_MAJOR_VERSION is explicitly set.

As a result, the Spark 4.1 starter runs using Spark 3's Scala 2.12 runtime, causing binary compatibility failures.

Resolution

Use the SeaTunnel Ambari MPack version that automatically configures SPARK_HOME and SPARK_MAJOR_VERSION in seatunnel-env.sh.

The MPack renders the following configuration from configuration/seatunnel-env.xml:

if [ -d "/usr/odp/current/spark4-client" ]; then
 export SPARK_HOME="${SPARK_HOME:-/usr/odp/current/spark4-client}"
 export SPARK_MAJOR_VERSION="${SPARK_MAJOR_VERSION:-4}"
fi

The Spark 4 starter loads seatunnel-env.sh through ${CONF_DIR}/seatunnel-env.sh, so no changes to the starter shell script are required.

For implementation details, see 11-odp343-stability-fixes.md, F-19.

20. Spark cluster-mode submission fails because the plugin archive is inaccessible or missing

Symptom

Submitting a SeaTunnel Spark job in YARN cluster mode fails because the plugin archive cannot be created or accessed.

The error depends on the installed SeaTunnel starter and MPack versions.

Scenario 1: AccessDeniedException in older builds

The job fails with an error similar to:

AccessDeniedException: /usr/odp/.../seatunnel/plugins.tar.gz

Scenario 2: FileNotFoundException after the F-21 workaround

The launcher creates the plugin archive in the temporary directory and registers it with deleteOnExit().

The submission process uses two JVMs:

1. The first JVM generates the spark-submit command and exits.
2. The second JVM executes the generated command and attempts to upload the plugin archive.

Because the first JVM deletes the archive when it exits, the second JVM fails with an error similar to:

java.io.FileNotFoundException:
File file:/tmp/seatunnel-plugins-<pid>-<ms>.tar.gz does not exist

The plugin archive lifecycle is not synchronized with the Spark submission process.

Earlier implementations either attempted to write the archive to a protected installation directory or deleted the temporary archive before Spark could upload it.

Resolution

The F-21 regression fix uses the canonical plugin archive:

$SEATUNNEL_HOME/plugins.tar.gz

The updated SparkStarter uses Common.pluginTarball() to reference the prebuilt archive instead of creating a temporary archive for each submission.

The source reports that the Spark 4 YARN W1 test completed successfully on October 9, 2026, with application ID application_1791554762409_0019.

20.1 Verify the plugin archive

1. Check whether the plugin archive exists:

sudo ls -la /usr/odp/current/seatunnel/plugins.tar.gz

2. 
   Verify the archive contents:

sudo tar tzf /usr/odp/current/seatunnel/plugins.tar.gz | head -3

3. Verify that the archive meets the expected requirements:
   • Ownership: root:hadoop
   • Permissions: 0644
   • Approximate size: 11 MB, depending on additional plugin JAR files
   • Archive contents: Files under the plugins/ directory

20.2 Check whether the Spark starter includes the fix

1. Set the starter JAR path:

JAR=/usr/odp/current/seatunnel/starter/seatunnel-spark-4.1-starter.jar

2. Extract the relevant class:

mkdir -p /tmp/xcheck && cd /tmp/xcheck && jar xf "$JAR" \
org/apache/seatunnel/core/starter/spark/'SparkStarter$ClusterModeSparkStarter.class'

3. Check whether the class contains deleteOnExit():

javap -c -p /tmp/xcheck/org/apache/seatunnel/core/starter/spark/'SparkStarter$ClusterModeSparkStarter.class' \
| grep -c deleteOnExit

The command returns 0 when the SeaTunnel RPM includes the F-21 regression fix.

A result of 1 indicates that the installed bytecode still contains the older archive deletion logic.

The source notes that ODP 3.4.3.0-1015 contains the older bytecode.

20.3 Rebuild the plugin archive if it is missing

If plugins.tar.gz is missing, restart the SEATUNNEL_ZETA_MASTER or SEATUNNEL_ZETA_WORKER component on the affected host through Ambari.

The archive builder runs from the following configuration methods:

• master.configure()
• worker.configure()
• client.configure()

20.4 Apply a temporary workaround for older builds

If the installed SeaTunnel RPM and MPack do not include both parts of the fix, create the plugin archive file manually:

sudo touch /usr/odp/current/seatunnel/plugins.tar.gz
sudo chown <submitting-user>:<submitting-user> /usr/odp/current/seatunnel/plugins.tar.gz
sudo chmod 664 /usr/odp/current/seatunnel/plugins.tar.gz

The permanent resolution requires:

• The Ambari MPack implementation of build_runtime_archives()
• The SeaTunnel SparkStarter update to use Common.pluginTarball()