Troubleshooting Spark 3

Issue: Service not visible in Ambari UI

Symptoms: Spark version doesn't appear in "Add Service" wizard

Solution:

# Verify mpack installation

ambari-server list-mpacks

​

# Expected output should include:

# spark3-ambari-3.5.5.mpack

​

# Clear Ambari cache

ambari-server stop

rm -rf /var/lib/ambari-server/cache/*

ambari-server start

Issue: History Server shows the wrong Spark version

Symptoms: Port 18083 shows Spark 3.5.5 instead of 3.3.3

Solution:

1. Verify configuration files exist:

ls -la /etc/spark3_3_3_3/conf/

2. Check port configuration:

grep "spark.history.ui.port" /etc/spark3_3_3_3/conf/spark-defaults.conf

3. Restart History Server from Ambari UI

Issue: spark-submit fails to find the correct version

Symptoms: Application uses the wrong Spark version

Solution:

Set environment variables explicitly:

export SPARK_HOME=/usr/odp/current/spark3_3_3_3-client export SPARK_CONF_DIR=/etc/spark3_3_3_3/conf /usr/odp/current/spark3_3_3_3-client/bin/spark-submit ...

Issue: Port conflict

Symptoms: History Server fails to start

Solution:

1. Check port availability:

netstat -tuln | grep 1808

2. Verify port configuration in Ambari
3. Ensure no other services use ports 18082-18084