Spark Live UI

Live Spark UI

Overview

The Live Spark UI provides a one-click link from a running job in xDP directly to the native Apache Spark web interface for that application. Use it to monitor active jobs, diagnose slow stages, and inspect executor resource usage in real time.

Accessing the Live Spark UI

Prerequisites

• An active compute cluster with Spark installed.

• A Spark job currently in the Running state.

• Appropriate permissions to view Spark job run details.

Steps

1. From the main navigation, go to Spark Jobs and click the job you want to monitor.

2. In the Job Runs table, click the Run ID of a run with status Running.

3. On the Run Details page, click Live Spark UI in the top-right action bar.

4. The native Apache Spark UI opens in a new browser tab.

The Spark UI is the standard Apache Spark web interface. It provides tabs for Jobs, Stages, Storage, Environment, Executors, and SQL / DataFrame — use these to monitor execution and resource usage in real time.

How-to Guides

How to Diagnose a Slow Stage

1. Open the Live Spark UI for the running job.

2. Go to the Stages tab and sort by Duration to find the slowest stage.

3. Click the stage's Description to open its detail page.

4. In the Tasks table, look for outliers in Duration, GC Time, or Shuffle Read/Write Size — these indicate data skew or resource pressure.

How to Check Executor Resource Usage

1. Open the Live Spark UI for the running job.

2. Go to the Executors tab.

3. Review Storage Memory, Disk Used, Cores, and Active Tasks for each executor.

4. Look for (dead) executors (node failure) or imbalanced Input / Shuffle values (partitioning issues).

Best Practices

• Correlate with cluster metrics. Pair Live Spark UI insights with infrastructure metrics from the Compute Clusters page for a complete picture.

• Use during development. Profiling jobs before production prevents costly failures and regressions.

• Switch to History Server post-run. Once a job finishes, use the Spark History Server for post-mortem analysis.