Troubleshooting Spark 3

Issue: Service not visible in Ambari UI

Symptoms: Spark version doesn't appear in "Add Service" wizard

Solution:

# Verify mpack installation
ambari-server list-mpacks

# Expected output should include:
# spark3-ambari-3.5.5.mpack
# Clear Ambari cache
ambari-server stop
rm -rf /var/lib/ambari-server/cache/*
ambari-server start

---

Issue: History Server shows the wrong Spark version

Symptoms: Port 18083 shows Spark 3.5.5 instead of 3.3.3

Solution:

• Verify configuration files exist:

ls -la /etc/spark3_3_3_3/conf/

• Check port configuration:

grep "spark.history.ui.port" /etc/spark3_3_3_3/conf/spark-defaults.conf

• Restart History Server from Ambari UI

---

Issue: spark-submit fails to find the correct version

Symptoms: Application uses the wrong Spark version

Solution:

Set environment variables explicitly:

export SPARK_HOME=/usr/odp/current/spark3_3_3_3-client export SPARK_CONF_DIR=/etc/spark3_3_3_3/conf /usr/odp/current/spark3_3_3_3-client/bin/spark-submit ...

---

Issue: Port conflict

Symptoms: History Server fails to start

Solution:

• Check port availability:

netstat -tuln | grep 1808

• Verify port configuration in Ambari
• Ensure no other services use ports 18082-18084