Troubleshooting Livy 3

Issue: Livy Server Not Visible in Ambari UI

Symptoms: Livy component doesn't appear when adding Spark service

Solution:

# Verify mpack installation
ambari-server list-mpacks

# Expected output should include spark3 mpacks
# spark3-ambari-3.5.5.mpack
# spark3-ambari-3.3.3.mpack
# Clear Ambari cache
ambari-server stop
rm -rf /var/lib/ambari-server/cache/*
ambari-server start

---

Issue: Port Conflict - "Address already in use"

Symptoms: Livy server fails to start with port binding error

Error Message:

Exception in thread "main" java.io.IOException: Failed to bind to /0.0.0.0:8333
Caused by: java.net.BindException: Address already in use

Solution:

• Check what's using the port:

sudo netstat -tulpn | grep 8333
# or
sudo lsof -i :8333

• Update Livy configuration:

# Edit configuration via Ambari UI:
# Services → SPARK3_3_3_3 → Configs → Advanced livy3-3.3.3-conf
# Change: livy.server.port = 8334 (or another available port)
# Or edit directly:
vi /etc/livy3_3_3_3/conf/livy.conf
# Update: livy.server.port = 8334

• Restart Livy Server:

# Via Ambari UI or:
ambari-agent restart

---

Issue: Wrong Spark Version Used

Symptoms: Livy sessions use incorrect Spark version

Solution:

• Verify SPARK_HOME in livy-env.sh:

# For Livy 3.3.3, should point to Spark 3.3.3
cat /etc/livy3_3_3_3/conf/livy-env.sh | grep SPARK_HOME
# Expected: export SPARK_HOME=/usr/odp/current/spark3_3_3_3-client

• Update via Ambari:
  • Services → SPARK3_3_3_3 → Configs → Advanced livy3-3.3.3-env
  • Set correct SPARK_HOME path
  • Save and restart Livy
• Verify Spark version in session:

# Create test session
SESSION_ID=$(curl -X POST http://localhost:8356/sessions \
 -H "Content-Type: application/json" \
 -d '{"kind":"spark"}' | jq -r '.id')
# Check Spark version
curl http://localhost:8356/sessions/$SESSION_ID | jq '.appInfo.sparkUiUrl'

---

Issue: Session Creation Fails

Symptoms: REST API returns error when creating sessions

Common Errors and Solutions:

# Check if Livy server is running

curl http://<host>:8333/version

# Check Livy logs
tail -f /var/log/livy333/livy-livy-server.out

[END_CODE_BLOCK]

{

  "conf": {
    "spark.yarn.queue": "existing-queue-name"
  }
}

[END_CODE_BLOCK]

# Check YARN capacity

yarn application -status

# Reduce resource requirements
curl -X POST http://<host>:8333/sessions \
 -H "Content-Type: application/json" \
 -d '{
    "kind": "spark",
    "conf": {
      "spark.executor.memory": "1g",
      "spark.executor.cores": 1,
      "spark.executor.instances": 1
    }
  }'

[END_CODE_BLOCK]

---

Issue: Sessions Timeout Unexpectedly

Symptoms: Livy sessions terminate before completion

Solution:

Increase session timeout:

# /etc/livy3_3_3_3/conf/livy.conf
livy.server.session.timeout = 2h
livy.server.session.timeout-check = true
livy.server.session.timeout-check.interval = 60000 # 60 seconds

Or disable timeout for long-running jobs:

livy.server.session.timeout = 0 # Disable timeout

---

Issue: Livy UI Shows Wrong Version

Symptoms: Livy REST API reports incorrect version

Solution:

• Verify correct binaries are being used:

ps aux | grep livy | grep java
# Should show correct path like /usr/odp/current/livy3_3_3_3-server

• Check symlinks:

ls -la /usr/odp/current/ | grep livy3

• Restart Livy with the correct LIVY_HOME:

export LIVY_HOME=/usr/odp/current/livy3_3_3_3-server
$LIVY_HOME/bin/livy-server restart

Session Pooling