Monitoring SeaTunnel — REST API, Web UI, Ambari, and logs

You can monitor a SeaTunnel deployment by using the Zeta REST API, Web UI, Ambari, logs, or JMX and Prometheus. Choose the monitoring method based on your requirements.

1. Zeta REST API

The SeaTunnel deployment uses two ports for different purposes. Do not use them interchangeably.

Port 8380 provides the following endpoints:

• /running-jobs
• /job-info/*
• /submit-job
• /stop-job
• /system-monitoring-information
• /finished-jobs/*

1.1 Check cluster health

Use the Hazelcast IMDG REST API on port 7654 to check cluster membership.

Run the following command:

curl -s http://<master>:7654/hazelcast/rest/cluster | jq

The response includes the complete member list and the liteMember flags.

In a healthy three-node cluster, the response includes six members:

• Three masters (full members)
• Three workers (lite members)

For a sample response, see section 2.2 in 01-overview.md.

Automated health check

Run the following command to check whether all six members are available:

EXPECTED=6 # 3 masters + 3 workers

ACTUAL=$(curl -s http://<master>:7654/hazelcast/rest/cluster | jq '.members | length')

[ "$ACTUAL" = "$EXPECTED" ] && echo OK || echo "DEGRADED: $ACTUAL/$EXPECTED members"

The command returns OK when all six members are available. Otherwise, it displays the number of available members.

Liveness check

Run the following command to check whether the Hazelcast member is active without retrieving the cluster membership information:

curl -s http://<master>:7654/hazelcast/health

The endpoint returns ACTIVE or PASSIVE.

1.2 Monitor running jobs

Use the Zeta REST API on port 8380 to retrieve running jobs.

Run the following command:

curl -s http://<master>:8380/running-jobs | jq

The response includes the following fields:

• jobId
• jobName
• jobStatus
• createTime
• jobDag

A job can transition through the following states:

SCHEDULED → RUNNING → CANCELING → FINISHED | FAILED | CANCELED

1.3 View job details

Run the following commands to retrieve the complete state of a specific job:

JOB_ID=1157808909784711184

curl -s http://<master>:8380/job-info/$JOB_ID | jq

The response includes per-vertex metrics, such as:

• metrics.SourceReceivedCount
• metrics.SinkWriteCount
• metrics.SinkWriteQPS

1.4 Monitor system metrics

Run the following command to retrieve CPU and memory metrics for each node:

curl -s http://<master>:8380/system-monitoring-information | jq '.[] | {host: .host, cpu, memory}'

Use these metrics to monitor resource utilization and plan cluster capacity.

1.5 Stop a running job

Run the following command to stop a running job:

curl -s -X POST "http://<master>:8380/stop-job" \

-H "Content-Type: application/json" \

-d "{\"jobId\": $JOB_ID, \"isStopWithSavePoint\": false}"

For streaming jobs, set isStopWithSavePoint to true to save the job state before stopping it.

To resume the job from its saved state, run the following command:

curl -s -X POST "http://<master>:8380/submit-job" \

-H "Content-Type: application/json" \

-d "{\"jobId\": $JOB_ID, \"restoreJobId\": $JOB_ID}"

1.6 Submit a HOCON job through REST

The /submit-job endpoint accepts HOCON configuration files when you specify:

• format=hocon in the query string.
• Content-Type: text/plain in the request header.

Do not use application/x-www-form-urlencoded. Jetty consumes the request body, causing the servlet to read an empty string.

The patched BaseServlet.getParameterMap() reads the query string directly to address this behavior.

For more information, see finding F-17 in 10-odp343-lab-findings.md.

Run the following command:

curl -s -X POST "http://<master>:8380/submit-job?format=hocon&jobName=my-hocon-job" \

-H "Content-Type: text/plain" \

--data-binary @/tmp/myjob.conf

2. Web UI

The SeaTunnel Zeta Web UI provides job monitoring information and a topology graph.

Access the Web UI at:

http://<master>:8380/

When HTTPS is enabled by setting enable_https=true, the configured endpoint is:

http://<master>:8483/

For HTTPS configuration details, see 06-security.md.

When SeaTunnel is configured behind Knox, access the Web UI at:

https://<knox>/gateway/<topology>/seatunnel/ui/

For more information, see the Knox section in 06-security.md.

You can also access the Web UI from Ambari by selecting the SeaTunnel Zeta Web UI quick link on the SEATUNNEL service summary card.

3. Ambari — Alerts, service state, and logs

Use Ambari to monitor SeaTunnel service status, configure alerts, and access logs.

3.1 Monitor service state

In Ambari, select SEATUNNEL to view the aggregated service state.

The service state can include:

• STARTED
• INSTALLED
• DECOMMISSIONING

Ambari determines the aggregated service state from individual host-component states.

Each host component uses the status() method in master.py or worker.py. These methods call SystemdManager.active(<unit>) to determine the service status.

Retrieve service state through REST

Run the following command:

curl -s -u admin:admin \

"http://<ambari>:8080/api/v1/clusters/<cluster>/services/SEATUNNEL?fields=ServiceInfo/state,alerts"

3.2 Configure built-in Ambari alerts

The ODP SeaTunnel management pack includes the following alerts.

The alert definitions are available in:

common-services/SEATUNNEL/2.3.13/alerts.json

To configure alert thresholds in Ambari, go to Alerts → SeaTunnel.

All three alerts use TCP port checks (PORT alert type). They do not monitor log-level or job-level health.

For more detailed monitoring, configure Prometheus to scrape the /system-monitoring-information endpoint on port 8380.

3.3 Use Ambari REST APIs

Retrieve host-component states

Run the following command:

curl -s -u admin:admin \

"http://<ambari>:8080/api/v1/clusters/<cluster>/services/SEATUNNEL/components?fields=host_components/HostRoles/state"

Restart SeaTunnel masters sequentially

Run the following command to restart a specified master through the Ambari REST API:

curl -s -u admin:admin -H "X-Requested-By: ambari" -X POST \

-d '{"RequestInfo":{"command":"RESTART","context":"rolling","operation_level":{"level":"HOST_COMPONENT",...}},

"Requests/resource_filters":[{"service_name":"SEATUNNEL","component_name":"SEATUNNEL_ZETA_MASTER","hosts":"<host>"}]}' \

"http://<ambari>:8080/api/v1/clusters/<cluster>/requests"

4. Logs — systemd, journald, and Log4j2

Use systemd, journald, and Log4j2 logs to monitor SeaTunnel services and troubleshoot failures.

4.1 Monitor logs in real time

Check the systemd service status

Run the following command to view the service status and the last 20 log lines:

sudo systemctl status seatunnel-master -n 20

Monitor journald logs

Run the following command to monitor the SeaTunnel master logs:

sudo journalctl -u seatunnel-master -f

systemd captures the standard output of the Bash and Java processes.

Monitor Log4j2 logs

Run the following commands to monitor the master and worker logs:

sudo tail -f /var/log/seatunnel/seatunnel-engine-master.log

sudo tail -f /var/log/seatunnel/seatunnel-engine-worker.log

4.2 Log file locations

Log rotation

Log4j2 manages rotation of the .log files based on the following settings:

• Daily rotation
• Maximum file size of 100 MB

The JVM manages rotation of the -gc.log* files.

The configuration does not use logrotate.

For more information, see section 11 in 05-troubleshooting.md.

4.3 Access logs from Ambari

If Log Search is deployed in the stack, go to:

Ambari → SEATUNNEL → Service Actions → Log Search

If Log Search is unavailable, use journalctl or monitor the log files directly.

5. JMX and Prometheus monitoring

The SeaTunnel master JVM supports JMX, but it does not expose an HTTP endpoint by default.

To enable JMX, configure jvm_tuning_profile in seatunnel-env to azul_prime or a custom profile, and add the following JVM options:

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.port=9091

-Dcom.sun.management.jmxremote.authenticate=false

-Dcom.sun.management.jmxremote.ssl=false

Then configure jmx_exporter as a sidecar, or expose the JMX port to Prometheus through a JMX-to-HTTP wrapper.

6. Monitor a job from submission to completion

The following examples demonstrate how to submit and monitor jobs on Zeta, Flink, and Spark.

6.1 Monitor a Zeta job

Step 1: Submit the job

Run the following command to submit a job and capture its job ID:

JOB_ID=$(sudo -u seatunnel /usr/odp/current/seatunnel/bin/seatunnel.sh --config myjob.conf 2>&1 | \

grep 'submit job' | grep -oE '[0-9]{19}')

Step 2: Monitor the job status

Run the following script to check the job status every five seconds:

while true; do

STATE=$(curl -s http://<master>:8380/job-info/$JOB_ID | jq -r .jobStatus)

echo "$(date '+%H:%M:%S') $JOB_ID $STATE"

case $STATE in FINISHED|FAILED|CANCELED) break;; esac

sleep 5

done

6.2 Monitor a Flink 1.19 job

Step 1: Submit the job

Run the following command:

APP_ID=$(sudo -u seatunnel bash -c "

kinit -kt /etc/security/keytabs/seatunnel.service.keytab seatunnel/\$(hostname -f)@\$REALM

/usr/odp/current/seatunnel/bin/start-seatunnel-flink-19-connector-v2.sh \

-e run-application --target yarn-application --config /tmp/myjob.conf 2>&1 | \

grep -oE 'application_[0-9_]+' | head -1

")

Step 2: Check the YARN application status

Run the following command:

yarn application -status $APP_ID

Step 3: Retrieve the application logs

Run the following command:

yarn logs -applicationId $APP_ID | grep -E "FINISHED|Total (Read|Write) Count"

6.3 Monitor a Spark 3.5 or Spark 4.1 job

Spark 3.5

Run the following command:

/usr/odp/current/seatunnel/bin/start-seatunnel-spark-3.5-connector-v2.sh \

--master yarn --deploy-mode cluster --config myjob.conf

Spark 4.1 (JDK 17)

Run the following command:

/usr/odp/current/seatunnel/bin/start-seatunnel-spark-4.1-connector-v2.sh \

--master yarn --deploy-mode cluster --config myjob.conf

After submitting the job, use the Spark History Server URL displayed in the command output to view the completed job details.

7. Validate Zeta high availability

The three-master Hazelcast topology supports the failure of a single master without interrupting cluster operations.

The following procedure validates high availability in a running cluster.

Step 1: Verify cluster membership

Run the following command:

curl -s http://<master>:7654/hazelcast/rest/cluster | jq '.members | length'

Expected result: Six members are available:

• Three full-member masters on port 7654
• Three lite-member workers on port 5801

Step 2: Stop one master

On the host that runs the master you want to stop, run the following command:

sudo systemctl stop seatunnel-master

Wait approximately 15 seconds for Hazelcast to detect that the member has left the cluster:

sleep 15

Step 3: Verify the remaining cluster members

Run the following command from any surviving master:

curl -s http://<other-master>:7654/hazelcast/rest/cluster | python3 -c "

import sys,json

d=json.load(sys.stdin)

mc=sum(1 for m in d['members'] if not m['liteMember'])

wc=sum(1 for m in d['members'] if m['liteMember'])

print('masters:', mc, 'workers:', wc)

for m in d['members']: print(' ', m['address'], 'lite='+str(m['liteMember']))

"

Expected result: Five members remain available:

• Two masters
• Three workers

Step 4: Submit a job while one master is unavailable

Run the following command:

sudo -u seatunnel bash -c '

source /etc/seatunnel/conf/seatunnel-env.sh

cd /usr/odp/current/seatunnel

./bin/seatunnel.sh --config /tmp/bench_w1.conf

'

Expected result: The job completes successfully even when one master is unavailable.

Step 5: Restart the master

Run the following commands:

sudo systemctl start seatunnel-master

sleep 15

Step 6: Verify that the master has rejoined the cluster

Run the following command:

curl -s http://<any-master>:7654/hazelcast/rest/cluster | jq '.members | length'

Expected result: The cluster membership count returns to six.

7.1 Lab validation results

The high-availability test was completed on October 1, 2026.

Before stopping the master

---- BEFORE ----

total: 6 masters: 3 workers: 3

After stopping master-1 and waiting 15 seconds

---- master-1 killed, 15 s wait ----

total: 5 masters: 2 workers: 3

[seatunnel-0.<fqdn>]:7654 full(master)

[seatunnel-2.<fqdn>]:7654 full(master)

[seatunnel-0.<fqdn>]:5801 lite(worker)

[seatunnel-1.<fqdn>]:5801 lite(worker)

[seatunnel-2.<fqdn>]:5801 lite(worker)

Job submission while master-1 was unavailable

A job containing one million rows was submitted through seatunnel-0.

---- Submitted W1 1M-row job via seatunnel-0 while master-1 down ----

Job (1157882837441970200) end with state FINISHED

Total Time(s): 7

Total Read Count: 1000000

Total Write Count: 1000000

Total Failed Count: 0

After restarting master-1

---- master-1 restarted + rejoined, 15 s ----

total: 6 (all 3 masters back + 3 workers)

7.2 Operational recommendations

The lab results confirmed the following behavior:

• The cluster remained operational when one of the three masters was unavailable.
• Jobs could be submitted through any surviving master.
• The restarted master rejoined the cluster and resumed its full-member role.
• The cluster retained a Hazelcast partition majority because at least two masters remained available.

To maintain high availability during maintenance, restart masters sequentially.

1. Restart one master at a time.
2. Wait until the Hazelcast membership count returns to six.
3. Restart the next master only after the previous master has rejoined.

This approach helps avoid the LiteNodeDropOutTcpIpJoiner 60-second blacklist race.

The management pack includes the following script to automate rolling restarts through the Ambari REST API:

docs/examples/rolling_restart.sh

The script uses a five-second interval between master restarts.