Acceldata
Acceldata XDP

Last updated: Nov 13, 2026 07:46 UTC

Spark Job Details

The Spark Job Details page provides detailed information about the execution and performance of an individual Spark application. Use this page to monitor resource utilization, analyze jobs and stages, investigate failures, and identify performance bottlenecks.

With xObserve, you can:

  • Troubleshoot failed or slow-running Spark applications.
  • Monitor job, stage, and task execution.
  • Analyze driver and executor resource utilization.
  • Identify inefficient stages, scheduling delays, and data skew.
  • Compare application runs to investigate performance differences.
  • Analyze execution plans and identify optimization opportunities.

Access Spark Job Details

To view the details of a Spark application:

  1. In the xObserve UI, navigate to Spark > Jobs.
  2. Select the cluster from the cluster selector in the upper-right corner.
  3. Locate the Spark application you want to investigate.
  4. Click the Application ID.

The Spark Job Details page opens, displaying application information, execution metrics, resource utilization, and performance analytics.

Spark application summary

The application summary provides an overview of the selected Spark application's execution status, resource utilization, and performance.

Application information

View the following application details:

  • Application ID
  • User
  • Final Status
  • Start Time
  • Duration
  • Queue
  • vCores and Memory

Execution statistics

Monitor the following metrics:

  • Number of Jobs
  • Number of Stages
  • Number of Tasks
  • Inefficient Stages
  • Blacklisted Stages
  • Average Memory Usage
  • Average vCore Usage
  • Scheduling Delay

Use these metrics to assess application execution and identify potential performance issues.

Job Trends

The Job Trends section displays historical execution trends for the same Spark application.

Monitor the following metrics:

  • Elapsed Time
  • vCore Usage
  • Memory Consumption
  • Input Data Read

Use this information to compare application runs, identify performance regressions, monitor workload growth, and evaluate tuning changes.

Compare Runs

Use Compare Runs to compare two Spark application executions and identify differences in performance, resource allocation, and configuration.

Select two Spark application IDs to view their execution details side by side.

Metric comparison

The Metric tab displays the following execution metrics:

  • Execution Time
  • Application ID
  • Start Time
  • End Time
  • Driver Cores
  • Executor Cores
  • Driver Memory
  • Executor Memory
  • Dynamic Executor Settings
  • Application Attempts

For each metric, the comparison displays the values from both applications and the percentage difference, where applicable.

Use this information to identify performance regressions, execution anomalies, and resource utilization differences.

Configuration comparison

The Config tab compares the Spark configurations used by the selected application runs.

Configuration details include:

  • Spark Application ID
  • Driver Host
  • Driver Address
  • Spark UI Proxy Settings
  • Application Start and Submit Times
  • JAR Locations
  • Driver and Executor Configuration Parameters
  • Spark Runtime Properties

Use this comparison to identify configuration changes that might affect application performance or execution behavior.

You can also use Compare Runs to validate tuning changes and troubleshoot differences between application executions.

Configuration details

The Configurations section displays the resources allocated to the Spark application.

View the following settings:

  • Number of Cores
  • Number of Executors
  • Executor Memory
  • Driver Cores
  • Driver Memory

Use these settings to investigate resource allocation and application performance.

Spark Jobs

The Spark Jobs section displays the jobs executed within the selected Spark application.

For each job, view:

  • Job ID
  • Job Name
  • Failed Stages
  • Completed Tasks
  • Duration
  • Status

You can also:

  • Use Group By Job ID to organize jobs by their IDs.
  • Use Show Inefficient Stages to identify stages with potential performance issues.
  • Expand a job to view its stages.

Job stages

Expand a job to view the following stage-level metrics:

  • Stage ID
  • Job ID
  • Task Count
  • Duration
  • Maximum Task Memory
  • I/O Utilization
  • Shuffle Read
  • Shuffle Write
  • Task Skew
  • Status
  • Failure Reason

Use these metrics to investigate long-running stages, data skew, shuffle bottlenecks, memory pressure, and execution failures.

Job Timeline

The Timeline view displays job execution activity chronologically.

Use this view to:

  • Track execution progress.
  • Identify scheduling delays.
  • Understand stage execution order.
  • Analyze parallel execution.

Timeseries Information

The Timeseries Information section displays runtime metrics for the selected Spark application.

Use the time range selector to focus on a specific execution period.

Schedule Information

The Schedule Information panel displays task scheduling metrics, including:

  • Active Tasks
  • Current Thread Pool Activity

Use these metrics to investigate scheduling delays, resource contention, and task concurrency.

You can also display metrics for individual executors.

I/O Metrics

The I/O section displays data read and write activity.

Monitor:

  • Input Bytes Read
  • Output Bytes Written

Use these metrics to analyze data throughput and identify I/O bottlenecks.

Driver Memory Usage

The Driver Memory Usage section displays memory utilization for the Spark driver.

Monitor:

  • Driver Total Memory
  • Heap Memory Usage Distribution

Use these metrics to identify memory pressure and evaluate driver memory allocation.

Executor Memory Usage

The Executor Memory Usage section displays memory utilization across Spark executors.

Monitor:

  • Executor Total Memory
  • Heap Memory Usage Distribution

You can view aggregated metrics or enable Show Individual Executors to analyze executor-specific memory usage.

Use this information to identify memory bottlenecks and evaluate executor resource allocation.

GC and CPU Distribution

The GC and CPU Distribution section displays JVM garbage collection and CPU utilization metrics.

Monitor:

  • JVM Garbage Collection Count
  • CPU Utilization

Use these metrics to identify excessive garbage collection, high CPU utilization, and resource inefficiencies.

You can also view executor-level metrics for detailed analysis.

Shuffle Information

The Shuffle Information section displays data transfer metrics between Spark stages.

Monitor:

  • Shuffle Bytes Written
  • Shuffle Local Bytes Read
  • Shuffle Remote Bytes Read
  • Shuffle Remote Bytes Read to Disk

Use these metrics to identify expensive shuffle operations, network bottlenecks, and data transfer overhead.

Storage Memory

The Storage Memory section displays Spark storage memory utilization.

Monitor:

  • Block Off-Heap Memory Used
  • Block On-Heap Memory Used
  • Block Max On-Heap Memory
  • Block Max Off-Heap Memory
  • Block Disk Space Used

Use these metrics to analyze caching behavior and storage memory consumption.

HDFS Information

The HDFS Information section displays storage activity associated with the Spark application.

Monitor:

  • HDFS Read Activity
  • HDFS Write Activity

Use these metrics to identify storage access patterns and potential bottlenecks.

Aggregate Metrics

The Aggregate Metrics table summarizes task execution statistics.

Metrics include:

  • Executor Runtime
  • JVM GC Time
  • Input Bytes Read
  • Disk Bytes Spilled
  • Memory Bytes Spilled

For each metric, the table displays:

  • Sum
  • Minimum
  • Maximum
  • Mean
  • Description

Use these metrics to identify resource-intensive tasks, excessive disk or memory spills, garbage collection overhead, and runtime inefficiencies.

Core Usage by Locality

The Core Usage by Locality section displays compute resource utilization.

Monitor:

  • Used Core Percentage
  • Wasted Core Percentage

Use these metrics to identify underutilized resources, evaluate executor sizing, and improve workload placement.

DAG Visualization

The DAG section displays the Spark execution plan as a Directed Acyclic Graph (DAG).

Use this visualization to understand execution flow, analyze stage dependencies, and identify expensive operations.

For each node, view details such as:

  • Operation Name
  • Duration
  • Output Rows
  • Execution Relationships

Use the DAG to investigate execution paths and identify optimization opportunities.

Additional actions

The Spark Job Details page also supports the following actions:

  • Search and filter job information, where available.
  • Expand jobs to view stage details.
  • Switch between Jobs and Timeline views.
  • Compare historical executions using Compare Runs.
  • Enable Show Individual Executors to view executor-specific metrics.
  • Refresh the page to retrieve the latest execution data.

This version preserves the original feature coverage and section flow while removing repetitive explanations, using action-oriented language, and keeping UI labels consistent.