Monitor SeaTunnel

SeaTunnel provides five methods for monitoring deployments: the Zeta REST API, Web UI, Ambari, logs, and JMX/Prometheus.

Use the monitoring method that best meets your operational requirements.

1. Zeta REST API

The Zeta REST API provides information about cluster health, running jobs, job status, and system metrics.

SeaTunnel uses two separate ports for cluster communication and job management.

Note: The /hazelcast/rest/metrics endpoint is not available in Hazelcast IMDG 5.1, which is included with SeaTunnel 2.3.13. This endpoint was available in Hazelcast 3.x but was removed in later versions. To retrieve system metrics, use the Zeta engine's :8380/system-monitoring-information endpoint or configure JMX. For more information, see 5. JMX and Prometheus Monitoring.

1.1 Check Cluster Health

Use the Hazelcast REST API on port 7654 to check cluster membership and health.

Run the following command:

curl -s http://<master>:7654/hazelcast/rest/cluster | jq

In a healthy three-node cluster, the response includes:

• Three master members (liteMember=false).
• Three worker members (liteMember=true).

For a sample response, see 01-overview.md, section 2.2.

Automate cluster health checks

Run the following commands to verify the expected number of cluster members:

EXPECTED=6   # 3 masters + 3 workers
ACTUAL=$(curl -s http://<master>:7654/hazelcast/rest/cluster | jq '.members | length')
[ "$ACTUAL" = "$EXPECTED" ] && echo OK || echo "DEGRADED: $ACTUAL/$EXPECTED members"

• OK if the cluster has all six expected members.
• DEGRADED if the number of members is fewer than expected.

Check cluster liveness

To check whether the Hazelcast member is active without retrieving cluster membership information, run:

curl -s http://<master>:7654/hazelcast/health

1.2 Monitor Running Jobs

Use the Zeta REST API on port 8380 to retrieve running jobs.

Run the following command:

curl -s http://<master>:8380/running-jobs | jq

• jobId: Unique identifier of the job.
• jobName: Name of the job.
• jobStatus: Current status of the job.
• createTime: Job creation time.
• jobDag: Directed acyclic graph (DAG) representing the job.

A job can transition through the following states:

SCHEDULED → RUNNING → CANCELING → FINISHED | FAILED | CANCELED

1.3 Retrieve Job Details

To retrieve the complete state of a specific job, run:

JOB_ID=1157808909784711184
curl -s http://<master>:8380/job-info/$JOB_ID | jq

• metrics.SourceReceivedCount: Number of records received by the source.
• metrics.SinkWriteCount: Number of records written to the sink.
• metrics.SinkWriteQPS: Sink write throughput.

1.4 Monitor System Metrics

To retrieve CPU and memory metrics for individual nodes, run:

curl -s http://<master>:8380/system-monitoring-information | jq '.[] | {host: .host, cpu, memory}'

1.5 Stop a Running Job

To stop a running job, run:

curl -s -X POST "http://<master>:8380/stop-job" \
     -H "Content-Type: application/json" \
     -d "{\"jobId\": $JOB_ID, \"isStopWithSavePoint\": false}"

To resume the job from its saved state, run:

curl -s -X POST "http://<master>:8380/submit-job" \

     -H "Content-Type: application/json" \

     -d "{\"jobId\": $JOB_ID, \"restoreJobId\": $JOB_ID}"

1.6 Submit a HOCON Job Using REST

The /submit-job endpoint supports Human-Optimized Config Object Notation (HOCON).

To submit a HOCON job, specify:

• format=hocon as a query parameter.
• Content-Type: text/plain as the request header.

Important: Do not use application/x-www-form-urlencoded as the content type. Jetty consumes the request body, causing the servlet to receive an empty string.

The patched BaseServlet.getParameterMap() method reads query parameters directly to address this behavior.

Run the following command:

curl -s -X POST "http://<master>:8380/submit-job?format=hocon&jobName=my-hocon-job" \
     -H "Content-Type: text/plain" \
     --data-binary @/tmp/myjob.conf

2. Web UI

The SeaTunnel Zeta Web UI provides a graphical interface for monitoring jobs and viewing their execution topology.

Access the Web UI at:

http://<master>:8380/

http://<master>:8483/

The Web UI displays the same job information available through the REST API and provides a topology graph.

Access the Web UI through Knox

When Knox is configured, access the Web UI at:

https://<knox>/gateway/<topology>/seatunnel/ui/

Access the Web UI through Ambari

On the Ambari SEATUNNEL service summary page, select the SeaTunnel Zeta Web UI quick link to open the Web UI.

3. Ambari — Alerts, Service State, and Logs

Ambari provides service-level monitoring, built-in alerts, and access to SeaTunnel logs.

3.1 Monitor Service State

To view the aggregated service state, navigate to Ambari > SEATUNNEL.

Ambari displays service states such as:

• STARTED
• INSTALLED
• DECOMMISSIONING

The aggregated service state is derived from the states of individual host components.

Each component's status() method in master.py or worker.py retrieves the state through SystemdManager.active(<unit>).

Retrieve service state using REST

Run the following command:

curl -s -u admin:admin \

  "http://<ambari>:8080/api/v1/clusters/<cluster>/services/SEATUNNEL?fields=ServiceInfo/state,alerts"

3.2 Monitor Built-in Ambari Alerts

The SeaTunnel Management Pack (MPack) includes the following alerts by default.

The alert definitions are available in:

common-services/SEATUNNEL/2.3.13/alerts.json

All three alerts use TCP port checks (PORT type). These alerts do not monitor job-level health or log-level events.

For more detailed monitoring, configure Prometheus to collect metrics from:

http://<master>:8380/system-monitoring-information

3.3 Use Ambari REST APIs

Ambari provides REST APIs for retrieving component states and managing service operations.

Retrieve the current state of all components

Run:

curl -s -u admin:admin \
  "http://<ambari>:8080/api/v1/clusters/<cluster>/services/SEATUNNEL/components?fields=host_components/HostRoles/state"

To restart a specific master component through the Ambari REST API, run:

curl -s -u admin:admin -H "X-Requested-By: ambari" -X POST \
  -d '{"RequestInfo":{"command":"RESTART","context":"rolling","operation_level":{"level":"HOST_COMPONENT",...}},
       "Requests/resource_filters":[{"service_name":"SEATUNNEL","component_name":"SEATUNNEL_ZETA_MASTER","hosts":"<host>"}]}' \
  "http://<ambari>:8080/api/v1/clusters/<cluster>/requests"

The MPack includes a rolling restart helper script at:

docs/examples/rolling_restart.sh

4. Logs — systemd, journald, and Log4j2

SeaTunnel uses systemd, journald, and Log4j2 to capture service lifecycle events, application logs, and task execution information.

4.1 View Logs in Real Time

Check systemd service status

To view the service status and the last 20 log lines, run:

sudo systemctl status seatunnel-master -n 20

To monitor logs captured from the Bash and Java processes by systemd, run:

sudo journalctl -u seatunnel-master -f

To monitor master and worker logs, run:

sudo tail -f /var/log/seatunnel/seatunnel-engine-master.log

sudo tail -f /var/log/seatunnel/seatunnel-engine-worker.log

4.2 Identify Log Locations

The following table lists the log types and their locations.

Log rotation

• Log4j2 rotates application log files daily or when they reach 100 MB.
• The JVM manages the rotation of GC log files (-gc.log*).
• SeaTunnel does not use logrotate for these logs.

For more information, see section 11 in 05-troubleshooting.md.

4.3 Access Logs Through Ambari

If Log Search is deployed in the stack, navigate to:

Ambari > SEATUNNEL > Service Actions > Log Search

If Log Search is not available, use journalctl or the log file commands described in 4.1 View Logs in Real Time.

5. JMX and Prometheus Monitoring

The SeaTunnel master Java Virtual Machine (JVM) supports Java Management Extensions (JMX). However, it does not expose an HTTP endpoint for JMX metrics by default.

To enable JMX monitoring:

1. Open the seatunnel-env configuration.
2. Set jvm_tuning_profile to azul_prime or use a custom profile.
3. Add the following JVM properties:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9091
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Important: The SeaTunnel MPack does not include the JMX exporter. Administrators must configure and manage the exporter separately. The example disables JMX authentication and SSL; restrict access to trusted networks or configure appropriate security controls before using it in production.

Note: The /hazelcast/rest/metrics endpoint was available in Hazelcast 3.x but was removed in Hazelcast IMDG 4.x and later. SeaTunnel 2.3.13 includes Hazelcast IMDG 5.1.

To retrieve per-node CPU and heap metrics, use:

http://<master>:8380/system-monitoring-information

6. Monitor a Specific Job End to End

The following examples demonstrate how to submit and monitor jobs using Zeta, Flink, and Spark.

6.1 Submit and Monitor a Zeta Job

Step 1: Submit the job

Run:

JOB_ID=$(sudo -u seatunnel /usr/odp/current/seatunnel/bin/seatunnel.sh --config myjob.conf 2>&1 | \
         grep 'submit job' | grep -oE '[0-9]{19}')

Step 2: Monitor job status

Run the following script to check the job status every five seconds:

# Poll every 5 s (job-level endpoints use port 8380, not 7654)
while true; do
  STATE=$(curl -s http://<master>:8380/job-info/$JOB_ID | jq -r .jobStatus)
  echo "$(date '+%H:%M:%S') $JOB_ID $STATE"
  case $STATE in FINISHED|FAILED|CANCELED) break;; esac
  sleep 5
done

• FINISHED
• FAILED
• CANCELED

Step 3: Retrieve completed job statistics

Run:

curl -s http://<master>:8380/finished-jobs/$JOB_ID | jq '{finishedTime, jobStatus, metrics}'

6.2 Submit and Monitor a Flink 1.19 Job

Step 1: Submit the job

Run:

APP_ID=$(sudo -u seatunnel bash -c "
  kinit -kt /etc/security/keytabs/seatunnel.service.keytab seatunnel/\$(hostname -f)@\$REALM
  /usr/odp/current/seatunnel/bin/start-seatunnel-flink-19-connector-v2.sh \
    -e run-application --target yarn-application --config /tmp/myjob.conf 2>&1 | \
  grep -oE 'application_[0-9_]+' | head -1
")

Run:

yarn application -status $APP_ID

Run:

yarn logs -applicationId $APP_ID | grep -E "FINISHED|Total (Read|Write) Count"

6.3 Submit and Monitor a Spark Job

SeaTunnel supports job submission using Spark 3.5 and Spark 4.1.

Submit a Spark 3.5 job

Run:

# Spark 3.5
/usr/odp/current/seatunnel/bin/start-seatunnel-spark-3.5-connector-v2.sh \
    --master yarn --deploy-mode cluster --config myjob.conf

Spark 4.1 requires JDK 17.

Run:

# Spark 4.1 (JDK 17)
/usr/odp/current/seatunnel/bin/start-seatunnel-spark-4.1-connector-v2.sh \
    --master yarn --deploy-mode cluster --config myjob.conf

7. Validate Zeta High Availability

A SeaTunnel cluster configured with three Hazelcast master members can continue operating if one master becomes unavailable.

Use the following procedure to validate high availability (HA) on a running cluster.

7.1 Verify Cluster Membership

Ensure that all three masters are running.

Run:

curl -s http://<master>:7654/hazelcast/rest/cluster | jq '.members | length'

6

• Three full-member masters on port 7654.
• Three lite-member workers on port 5801.

7.2 Stop One Master and Verify Cluster Availability

Step 1: Stop a master

On the host where you want to stop the master, run:

sudo systemctl stop seatunnel-master

Allow approximately 15 seconds for Hazelcast to detect that the member has left the cluster.

sleep 15

Run the following command on a surviving master:

curl -s http://<other-master>:7654/hazelcast/rest/cluster | python3 -c "
import sys,json
d=json.load(sys.stdin)
mc=sum(1 for m in d['members'] if not m['liteMember'])
wc=sum(1 for m in d['members'] if m['liteMember'])
print('masters:', mc, 'workers:', wc)
for m in d['members']: print(' ', m['address'], 'lite='+str(m['liteMember']))
"

• Two masters.
• Three workers.
• Five members in total.

7.3 Submit a Job While One Master Is Unavailable

To verify that the cluster continues accepting jobs, submit a job through a surviving master.

Run:

sudo -u seatunnel bash -c '
  source /etc/seatunnel/conf/seatunnel-env.sh
  cd /usr/odp/current/seatunnel
  ./bin/seatunnel.sh --config /tmp/bench_w1.conf
'

7.4 Restore the Master and Verify Cluster Membership

Step 1: Start the stopped master

Run:

sudo systemctl start seatunnel-master

Run:

sleep 15

Run:

curl -s http://<any-master>:7654/hazelcast/rest/cluster | jq '.members | length'

6

7.5 Lab Validation Results

The following results were recorded during HA validation on October 1, 2026.

Before stopping a master

---- BEFORE ----
total: 6  masters: 3  workers: 3

---- master-1 killed, 15 s wait ----
total: 5  masters: 2  workers: 3
  [seatunnel-0.<fqdn>]:7654 full(master)
  [seatunnel-2.<fqdn>]:7654 full(master)
  [seatunnel-0.<fqdn>]:5801 lite(worker)
  [seatunnel-1.<fqdn>]:5801 lite(worker)
  [seatunnel-2.<fqdn>]:5801 lite(worker)

---- Submitted W1 1M-row job via seatunnel-0 while master-1 down ----
Job (1157882837441970200) end with state FINISHED
Total Time(s):   7
Total Read Count: 1000000
Total Write Count: 1000000
Total Failed Count: 0

---- master-1 restarted + rejoined, 15 s ----
total: 6  (all 3 masters back + 3 workers)

• The cluster remained operational when one of the three masters became unavailable.
• Jobs could be submitted through a surviving master.
• The stopped master successfully rejoined the cluster after restarting.
• Hazelcast maintained a majority with two of the three masters available.

7.6 Rolling Restart Recommendations

To maintain cluster availability during rolling restarts:

1. Restart one master at a time.
2. Wait for the restarted master to rejoin the Hazelcast cluster.
3. Verify that cluster membership returns to six members.
4. Restart the next master only after the previous master has rejoined.

This approach helps prevent the LiteNodeDropOutTcpIpJoiner 60-second blacklist race.

The MPack includes the following rolling restart helper script:

docs/examples/rolling_restart.sh

Important: For a three-master Hazelcast cluster, at least two masters must remain available to maintain a majority. Avoid restarting multiple masters simultaneously.