Spark History Server

Overview

The Spark History Server provides a persistent web UI for analyzing completed Spark applications. Use it to debug failures, troubleshoot performance bottlenecks, and review resource utilization long after a job has finished running.

Accessing the Spark History Server

Prerequisites

• A completed (Success or Failed) Spark job in xDP.
• Appropriate permissions to view job run details.

Steps

1. From the xDP side navigation, go to Spark > Job Runs.
2. Select a completed job run to open its details page.
3. In the top-right corner of the Run ID details page, click Spark History Server.
4. You are redirected to an authentication portal. Click Sign in with OpenID Connect to proceed.
5. After authenticating, the standard Apache Spark UI opens.

The Spark UI is the standard Apache Spark web interface. It provides tabs for Jobs, Stages, Storage, Environment, Executors, and SQL / DataFrame — use these to investigate job execution, resource usage, and query plans as needed.

How-to Guides

How to Identify a Performance Bottleneck

1. Open the Spark History Server for the target job run.
2. Navigate to the Stages tab and sort by Duration to find the longest-running stage.
3. Click the stage's Description link to open its detail page.
4. In the Tasks table, sort by Duration to find straggler tasks.
5. Check for high GC Time, Shuffle Write Time, or disproportionate Input Size (data skew) among the slowest tasks.

How to Verify Spark Configuration for a Past Job Run

1. Open the Spark History Server for the job run.
2. Navigate to the Environment tab.
3. Scroll to the Spark Properties table.
4. Use browser search (Ctrl+F / Cmd+F) to locate specific properties such as spark.executor.memory or spark.sql.shuffle.partitions.

Best Practices

• Right-size your resources. Use the Executors tab to check memory and GC time. Consistently low storage memory suggests over-provisioning; high GC time suggests under-provisioning.
• Review critical jobs proactively. Periodically inspect the History Server for long-running pipelines to catch performance regressions early.
• Correlate with cluster metrics. Combine History Server data with infrastructure metrics from the xDP Compute Clusters page for a complete picture.