SeaTunnel FAQs

This section answers frequently asked questions about SeaTunnel architecture, state management, execution engines, resource management, and migration from Sqoop.

1. Where does SeaTunnel store its state and metadata? Does it require an external database?

SeaTunnel does not require an external database to store its state and metadata when using the Zeta engine.

The Zeta engine stores live state in the embedded Hazelcast in-memory data grid (IMDG) cluster, which runs within each Zeta master Java Virtual Machine (JVM).

State and metadata storage

The following table describes the Hazelcast maps used to store state and metadata.

Data persisted outside the JVM

SeaTunnel persists the following data outside the JVM:

Checkpoint data

• Applies only to streaming jobs.
• Uses the state.checkpoints.dir property in seatunnel.yaml.
• The default location on this stack is HDFS: hdfs:///user/seatunnel/flink-checkpoints.
• Stores exactly-once, barrier-aligned snapshots.

Hazelcast Hot Restart

• Provides optional local disk persistence for IMDG partitions.
• Is disabled by default.
• Is configured under the persistence section in hazelcast-master.yaml.
• Is intended primarily for IMDG cache recovery after master restarts, rather than Zeta job state recovery.

SeaTunnel does not require a relational database management system (RDBMS), ZooKeeper, or an Amazon S3 metadata bucket for this state management.

The following REST endpoints retrieve information directly from Hazelcast IMaps:

• /running-jobs
• /finished-jobs/*
• /job-info/{id}

What happens if a Zeta master JVM fails?

If a Zeta master JVM fails while Hazelcast Hot Restart is disabled:

• Batch jobs: Restart from the beginning.
• Streaming jobs: Resume from the last successful checkpoint stored in HDFS.

For more information about the REST endpoints, see Section 1 in 04-monitoring.md.

2. What capabilities does Zeta provide that the Flink and Spark starters do not?

The Zeta engine provides several capabilities that simplify job execution, resource management, and high availability.

Although Flink also supports change data capture (CDC), Zeta provides operational advantages for certain workloads.

Multi-table synchronization

Zeta supports synchronizing multiple tables within a single job.

For example, a source configuration can specify:

table-names = ["db.t1", "db.t2"]

With the Flink and Spark starters, separate jobs are required for each table.

Schema evolution without restarting jobs

Zeta uses TableSchemaEvolveListener to detect data definition language (DDL) changes in CDC sources and propagate supported schema changes to sinks while jobs are running.

• ADD COLUMN: Supported.
• DROP COLUMN and RENAME COLUMN: Supported with limitations.

REST-based job lifecycle management

Zeta provides REST endpoints to manage and monitor jobs.

These operations interact directly with Hazelcast IMaps and use a consistent job ID throughout the job lifecycle.

Flink also provides REST APIs. However, integrating SeaTunnel job management with Flink requires mapping SeaTunnel job identities to Flink job identities.

Resource groups using worker tags

Zeta supports assigning specific job categories to designated worker nodes by using tag.restrictions in seatunnel.yaml.

For example, you can assign resource-intensive CDC jobs to dedicated workers to prevent Kafka traffic spikes from affecting Hive sink jobs.

YARN provides queue-based resource management but does not provide the same job-level worker tag matching.

Lite workers

Zeta supports workers configured as lite-member.

These workers execute tasks without owning Hazelcast data partitions, reducing heap usage on nodes dedicated to task execution.

This worker configuration is specific to Zeta and does not have a direct equivalent in the Flink or Spark starters.

Faster job startup

Zeta masters run continuously, allowing jobs to be submitted through Hazelcast remote procedure calls (RPCs).

Typical startup times described for this deployment are:

The shorter startup time makes Zeta suitable for frequently executed, short-lived jobs.

Simplified high availability

Zeta supports high availability (HA) using three master nodes without requiring an additional ZooKeeper service.

Flink HA requires ZooKeeper, HDFS, and additional configuration.

Limitations of Zeta

Consider the following limitations when selecting an execution engine:

• Large streaming state: Flink's checkpoint coordinator and RocksDB state backend are more established for workloads with streaming state exceeding 100 TB.
• Complex analytical queries: Spark's Catalyst optimizer provides advantages when planning complex Spark SQL queries.

3. Does Zeta spill data to disk? What happens when a job exceeds the JVM heap?

Zeta does not automatically spill row data to disk in the same way as Flink's RocksDB state backend or Spark's shuffle and broadcast spill mechanisms.

Runtime buffers reside in the Zeta JVM heap, including:

• Source batches
• Intermediate transformation data
• Sink accumulators

Hazelcast IMaps used for checkpoint state also reside in memory, with periodic persistence to the directory configured by state.checkpoints.dir.

Behavior for streaming jobs

For streaming jobs, Zeta uses backpressure to regulate data flow.

When a sink cannot process incoming data quickly enough:

1. The Hazelcast queue between the source and sink fills.
2. Zeta's credit-based flow control applies backpressure.
3. The source reduces its data production rate.
4. The pipeline continues processing at a reduced rate.

This mechanism helps prevent out-of-memory (OOM) errors caused by queue growth.

The memory limit is determined by:

taskmanager.memory.network.max × parallelism × workers

Behavior for batch jobs

For batch jobs, if a transformation's working set exceeds the available JVM heap, the Zeta worker can encounter an OOM error, causing the job to fail.

Zeta does not automatically spill this data to disk.

Recommended workarounds

Use the following approaches in the specified order:

1. Increase source partitioning. Increase partition_num for the JDBC source so that each reader processes a smaller data range.
2. Add more workers. Scale Zeta horizontally to increase aggregate heap capacity. For example, three workers provide approximately three times the aggregate heap capacity of one worker with the same configuration.
3. Increase JVM heap size. Configure seatunnel_master_heap_mb and seatunnel_worker_heap_mb through Ambari.
4. Use Flink or Spark. Select an execution engine that supports disk spilling for memory-intensive workloads.

For example, Spark 3.5 or 4.1 is more suitable than Zeta for a MySQL-to-Hive batch transfer containing 500 million rows.

Engine selection recommendations

• Use Zeta for small-to-medium batch workloads, typically fewer than 50 million rows per worker.
• Use Zeta for streaming workloads where throughput is constrained by sink processing capacity.
• Use Spark or Flink for large batch shuffles, complex joins, broadcast-intensive transformations, or workloads where state exceeds the available node memory.

For more information about heap configuration, see Section 13, Tuning, in 05-troubleshooting.md.

4. What are the differences between SeaTunnel and Sqoop, and when should each be used?

SeaTunnel and Sqoop support data ingestion and transfer, but they differ in connector availability, execution engines, streaming capabilities, and deployment requirements.

Feature comparison

When should you use SeaTunnel?

SeaTunnel is suitable for the following scenarios:

CDC workloads

Use SeaTunnel to continuously capture changes from supported databases, including MySQL, PostgreSQL, Oracle, and SQL Server.

Non-JDBC data sources

SeaTunnel supports data pipelines such as:

• Kafka to supported sinks
• Amazon S3 to supported sinks
• MongoDB to supported sinks

Modern data lakehouse integrations

SeaTunnel supports sinks such as Iceberg v2, Hudi Merge on Read (MOR), and Delta.

Sqoop can write data in formats such as Parquet and Avro but does not directly manage the metadata catalogs associated with these lakehouse formats.

Unified batch and streaming pipelines

SeaTunnel supports configuring batch and streaming execution through HOCON by changing the job.mode setting.

In-flight data transformations

SeaTunnel supports field-level transformations during pipeline execution.

Sqoop typically requires additional processing through tools such as Hive.

Multi-table synchronization

SeaTunnel can synchronize multiple tables within a single Zeta job, whereas Sqoop requires separate jobs.

When should you continue using Sqoop?

Sqoop remains suitable for the following scenarios:

Simple JDBC-to-HDFS transfers

For straightforward JDBC-to-HDFS Parquet transfers, Sqoop provides a simpler CLI-based workflow without requiring a HOCON configuration file and additional Kerberos settings.

Existing YARN deployments

Sqoop runs as a CLI tool on an existing YARN cluster.

SeaTunnel with Zeta requires additional master and worker JVMs that must be monitored, maintained, and upgraded.

You can reduce this overhead by using SeaTunnel in client-only mode with the Flink or Spark starters.

Database-specific type compatibility

Sqoop includes years of compatibility improvements for database-specific types, including:

• Oracle BLOB
• SQL Server datetimeoffset
• PostgreSQL BIT
• MySQL BIT(1) to Boolean conversions

SeaTunnel's JDBC connector supports most common data types but has limitations. For example, PostgreSQL JSONB values can be converted to strings.

Existing operational integrations

Continue using Sqoop when existing pipelines depend on tools such as Oozie, Autosys, cron, or custom monitoring scripts that parse Sqoop output.

Migrating stable pipelines might introduce unnecessary operational effort without providing additional business value.

Legacy MapReduce clusters

Sqoop remains an option for Hadoop 2.x clusters that use MapReduce without Spark or Flink.

SeaTunnel does not provide a MapReduce execution engine.

Recommendation

Use SeaTunnel for new data pipelines, particularly those involving streaming, CDC, non-JDBC sources, or modern lakehouse integrations.

Continue using Sqoop for stable, legacy JDBC bulk-transfer workloads unless there is a specific business requirement to migrate.

5. How can a Sqoop-to-SeaTunnel converter tool work?

A Sqoop-to-SeaTunnel converter translates Sqoop CLI commands into SeaTunnel HOCON configuration files.

High-level architecture

The conversion process follows this sequence:

Sqoop CLI command
       |
       v
     Parser
       |
       v
Canonical intermediate representation (IR)
       |
       v
  HOCON generator
       |
       v
SeaTunnel .conf file

Sqoop argument mapping

The following table describes how Sqoop arguments can be mapped to SeaTunnel configuration properties.

When converting Hive imports, the generated configuration must also include the required settings:

• metastore_uri
• hive_site_path
• hdfs_site_path
• Kerberos configuration

Managing incremental state

Sqoop stores incremental watermarks in a Derby metastore when using sqoop job.

A converter must provide an alternative mechanism to manage incremental state.

Consider the following approaches.

Option A: Stateless watermark management

Generate SeaTunnel configurations that accept a watermark through an environment variable.

${WATERMARK}

This approach is simple but does not preserve Sqoop's automatic watermark advancement.

Option B: Watermark file

Generate a shell script that reads and updates a watermark file before and after SeaTunnel execution.

This approach reproduces Sqoop's state-management behavior but introduces additional filesystem state.

Option C: Query the target system

Retrieve the current watermark from the target before executing the generated SeaTunnel job.

For example:
SELECT MAX(check_column) FROM target;

This approach requires a queryable target system.

The lab example demo_04_incremental_load.sh demonstrates this pattern.

Recommended approach:

• Use Option C for Hive and Iceberg sinks.
• Use Option A for Kafka and file sinks.
• Use Option B when the other approaches are not suitable.

Validating converted jobs

Validate the converter by comparing the output of the original Sqoop job with the converted SeaTunnel job.

1. Run the original Sqoop job against a test MySQL database and HDFS.
2. Capture the job output.
3. Run the converted SeaTunnel configuration against the same source and destination.
4. Capture the SeaTunnel output.
5. Compare the results for:
   • Row counts
   • Schema using parquet-tools
   • Null-value handling
   • Data type conversions
   • File layout

A converter that performs only syntax mapping can miss approximately 15% of cases involving null handling, data type compatibility, string encoding, and unsupported direct-mode options.

Classify unsupported configurations as unsupported — manual review in the converter output.

Suggested implementation technologies

The estimated implementation effort is:

• Approximately 300 lines of Python for the parser and mapping framework.
• Approximately 200 lines of Python for the validation component.

6. Can SeaTunnel run without Zeta in client-only mode, similar to Sqoop?

Yes. SeaTunnel can run in client-only mode using the Flink or Spark starters without requiring a running Zeta cluster.

How does client-only execution work with Flink?

Consider the following command:

start-seatunnel-flink-2-connector-v2.sh --config x.conf -t yarn-application

1. The SeaTunnel starter JAR acts as a lightweight CLI wrapper.
2. The starter reads the HOCON configuration file.
3. SeaTunnel generates a Flink DataStream graph using seatunnel-translation-flink-2.
4. The starter submits the graph through flink run --target yarn-application.
5. The Flink client communicates with the YARN ResourceManager (RM), allocates the ApplicationMaster (AM) and TaskManagers (TMs), and distributes the graph and application code.
6. The SeaTunnel CLI exits after the Flink client returns.

No SeaTunnel-specific JVM remains running after submission.

How does client-only execution work with Spark?

The Spark 3.5 and 4.1 starters follow a similar process.

1. The SeaTunnel starter reads the HOCON configuration.
2. The starter generates the Spark execution graph.
3. The starter submits the job through spark-submit.
4. The SeaTunnel starter JVM exits after submission.
5. The Spark driver and executors continue running on YARN.

In this execution mode, SeaTunnel functions as a client-only tool, similar to Sqoop.

The SeaTunnel Zeta service does not need to be running for Flink or Spark job execution.

What does the Ambari management pack provide without Zeta?

Even when Zeta is not used, the Ambari management pack (mpack) provides the following components and configurations.

The environment configuration also includes the Java 17-dependent Spark 4 configuration block for cross-ODP compatibility.

The odp-select integration uses:

stack_select.select("seatunnel-server", version)

What are the limitations of client-only deployments?

The current Ambari mpack deploys master and worker service components even when only the Flink or Spark starters are required.

You cannot install SeaTunnel through Ambari without registering these components.

For client-only deployments, use the following configuration approach:

1. Install SeaTunnel through Ambari.
2. Set master_heap_mb and worker_heap_mb to their minimum values of 512 MB each.
3. Stop the SeaTunnel master and worker services after installation to avoid unnecessary resource consumption.
4. Run jobs only through the Flink or Spark starter scripts:
5. 
   start-seatunnel-{flink,spark}-connector-v2.sh
6. 
   Do not use seatunnel.sh or seatunnel-cluster.sh.
7. Disable the corresponding Ambari port alerts, including seatunnel_master_hazelcast_port, to prevent unnecessary alerts for intentionally stopped services.

Potential enhancement: Dedicated SeaTunnel client role

A dedicated SEATUNNEL_CLIENT component in metainfo.xml could simplify client-only deployments.

The component would:

• Use the CLIENT component type.
• Provide install() and configure() methods.
• Exclude start() and stop() methods.
• Follow the same design pattern as HDFS_CLIENT, YARN_CLIENT, and SPARK3_CLIENT.

This component is not currently available.

Adding a dedicated client role is a potential enhancement for environments that use only Flink or Spark execution.

The estimated implementation effort is approximately half a day.

Summary: SeaTunnel execution modes

Key takeaway: When Zeta is enabled, SeaTunnel operates as a distributed ETL and CDC platform with its own execution, state management, HA, and REST interfaces.

When using only the Flink or Spark starters, SeaTunnel acts as a configuration-driven job generator. The underlying execution engine manages the job, and the Zeta services do not need to remain running.