Monitor Spark

Apache Spark is a distributed data processing engine used for large-scale analytics and data processing. It provides fast, scalable, and fault-tolerant execution for batch and streaming workloads across clustered environments.

Observability with xObserve

Monitor Spark applications deployed on your xCompute cluster through xObserve to:

• Track overall Spark workload activity and cluster health.
• Monitor resource utilization, including CPU and memory allocation.
• Analyze Spark job execution, performance, and status.
• Investigate individual job details for troubleshooting and optimization.
• Identify workload trends and user activity across the cluster.
• Detect failures and performance bottlenecks proactively.

Prerequisites

To enable Spark observability:

• Enable the xObserve Integration toggle for the Spark application in the XDP UI.
• You can enable this option:
  • During Spark installation, or
  • By clicking Edit on an existing Spark application.

Once enabled, Spark metrics become available in xObserve.

Navigation Steps

1. In the XDP UI, navigate to Infrastructure from the left navigation pane.
2. Select Apps.
3. On the Apps page, locate the Spark application and click the xObserve icon.
4. The Spark application dashboard opens in xObserve, where you can view application metrics, performance insights, and health information.

Explore In Detail

• Spark Dashboard
• Spark Jobs
• Spark Job Details