The Spark Job Details page provides comprehensive visibility into the execution and performance of an individual Spark application. It enables administrators and data engineers to investigate application behavior, analyze resource utilization, monitor stage and task execution, and identify performance bottlenecks.
With xObserve, you can:
- Troubleshoot failed or slow-running Spark applications.
- Identify inefficient stages and execution bottlenecks.
- Analyze driver and executor resource utilization.
- Monitor stage, task, and job execution behavior.
- Compare application runs to understand performance differences.
- Investigate scheduling delays and workload inefficiencies.
- Gain insights into Spark execution plans and optimization opportunities.
Navigation Steps
- In the xObserve UI, navigate to Spark > Jobs from the left navigation pane.
- Select the desired cluster from the cluster selector in the upper-right corner.
- Locate the Spark application you want to investigate.
- Click the Application ID.
The Spark Job Details page opens, displaying execution metrics, resource utilization, job breakdowns, timelines, and performance analytics for the selected Spark application.
Spark Application Summary
The top section provides a high-level overview of the Spark application execution.
Application Information
View key execution details such as:
- Application ID
- User
- Final Status
- Start Time
- Duration
- Queue
Execution Statistics
Monitor execution-level metrics including:
- Number of Jobs
- Number of Stages
- Number of Tasks
- Inefficient Stages
- Blacklisted Stages
- Average Memory Usage
- Average vCore Usage
- Scheduling Delay
Use these metrics to understand workload scale and overall execution health quickly.
Job Trends
The Job Trends section compares historical executions of the same Spark application.
Track metrics such as:
- Elapsed Time
- vCore Usage
- Memory Consumption
- Input Data Read
Use this view to:
- Compare current execution against previous runs
- Identify performance regressions
- Detect workload growth trends
- Validate tuning improvements
Compare Runs
Use Compare Runs to compare two Spark application executions and identify differences in performance, resource utilization, and runtime configuration.
The comparison view allows you to select two Spark application IDs and analyze their execution characteristics side by side.
Metric Comparison
The Metric tab compares key execution metrics between the selected application runs, including:
- Execution Time
- Application ID
- Start Time
- End Time
- Driver Cores
- Executor Cores
- Driver Memory
- Executor Memory
- Dynamic Executor Settings
- Application Attempts
For each metric, the comparison view displays:
- Left Application Value
- Right Application Value
- Percentage Difference (when applicable)
Use this view to identify performance regressions, execution anomalies, and resource utilization differences across runs.
Configuration Comparison
The Config tab compares Spark configuration properties used by each application execution.
Examples include:
- Spark Application ID
- Driver Host
- Driver Address
- Spark UI Proxy Settings
- Application Start and Submit Times
- JAR Locations
- Driver and Executor Configuration Parameters
- Spark Runtime Properties
Configuration comparison helps identify changes that may have impacted performance, execution behavior, or resource consumption.
Use Cases
Use Compare Runs to:
- Investigate performance regressions.
- Validate Spark tuning changes.
- Compare resource allocation between executions.
- Analyze the impact of configuration modifications.
- Troubleshoot differences in execution behavior across application runs.
Configuration Details
The Configurations section displays the Spark resources allocated to the application.
Configuration information includes:
- Number of Cores
- Number of Executors
- Executor Memory
- Driver Cores
- Driver Memory
Use this information when analyzing performance issues related to resource sizing or workload tuning.
Spark Jobs
The Spark Jobs section provides a breakdown of all jobs executed within the Spark application.
For each job, view:
- Job ID
- Job Name
- Failed Stages
- Completed Tasks
- Duration
- Status
Additional options include:
- Group By Job ID
- Show Inefficient Stages
These views help identify problematic or resource-intensive jobs.
Job Stages
Expand a job to view its stage-level details.
For each stage, monitor:
- Stage ID
- Job ID
- Task Count
- Duration
- Maximum Task Memory
- I/O Utilization
- Shuffle Read
- Shuffle Write
- Task Skew
- Status
- Failure Reason
Use stage-level metrics to identify:
- Long-running stages
- Data skew issues
- Shuffle bottlenecks
- Memory pressure
- Failed execution stages
Job Timeline
The Timeline view provides a chronological representation of job execution activity.
Use the timeline to:
- Visualize execution progress
- Identify scheduling delays
- Understand stage sequencing
- Analyze parallel execution behavior
Time Series Information
The Timeseries Information section displays runtime trends for the selected Spark application.
Track metrics such as:
- Driver Time
- Executor Activity
- Resource Utilization Trends
Use the time range selector to focus on specific execution windows.
Charts
Schedule Information
The Schedule Information panel helps analyze task scheduling behavior.
Metrics include:
- Active Tasks
- Current Thread Pool Activity
This view helps identify:
- Scheduling delays
- Resource contention
- Task concurrency behavior
You can optionally display metrics for individual executors.
I/O Metrics
The I/O section provides visibility into application data movement.
Monitor:
- Input Bytes Read
- Output Bytes Written
Use these metrics to understand workload throughput and identify I/O bottlenecks.
Driver Memory Usage
Monitor memory utilization of the Spark driver process.
Metrics include:
- Driver Total Memory
- Heap Memory Usage Distribution
Use this information to:
- Detect memory pressure
- Prevent driver out-of-memory failures
- Optimize driver sizing
Executor Memory Usage
Track memory consumption across Spark executors.
Metrics include:
- Executor Total Memory
- Heap Memory Usage Distribution
You can view aggregate metrics or display individual executor statistics.
Use this information to:
- Identify executor memory bottlenecks
- Detect inefficient memory utilization
- Tune executor resource allocation
GC and CPU Distribution
Analyze JVM and CPU behavior during execution.
Metrics include:
- JVM Garbage Collection Count
- CPU Utilization
Use these metrics to identify:
- Excessive garbage collection activity
- CPU saturation
- Resource inefficiencies
Executor-level views are available for deeper analysis.
Shuffle Information
The Shuffle Information section provides insights into data exchange between Spark stages.
Monitor:
- Shuffle Bytes Written
- Shuffle Local Bytes Read
- Shuffle Remote Bytes Read
- Shuffle Remote Bytes Read to Disk
Use these metrics to:
- Detect expensive shuffle operations
- Identify network bottlenecks
- Analyze stage communication overhead
Storage Memory
Monitor Spark storage memory utilization.
Metrics include:
- Block Off-Heap Memory Used
- Block On-Heap Memory Used
- Block Max On-Heap Memory
- Block Max Off-Heap Memory
- Block Disk Space Used
Use this information to evaluate caching effectiveness and memory consumption patterns.
HDFS Information
The HDFS Information section provides visibility into storage interactions.
Monitor:
- HDFS Read Activity
- HDFS Write Activity
Use these metrics to understand storage access patterns and identify potential storage bottlenecks.
Aggregate Metrics
The Aggregate Metrics table provides summarized execution statistics across tasks.
Examples include:
- Executor Runtime
- JVM GC Time
- Input Bytes Read
- Disk Bytes Spilled
- Memory Bytes Spilled
For each metric, view:
- Sum
- Minimum
- Maximum
- Mean
- Description
Use aggregate metrics to identify:
- Resource-intensive workloads
- Spill-heavy executions
- GC overhead
- Runtime inefficiencies
Core Usage by Locality
The Core Usage by Locality section helps evaluate compute efficiency.
Metrics include:
- Used Core Percentage
- Wasted Core Percentage
Use this view to:
- Identify underutilized resources
- Improve executor sizing
- Optimize workload placement
- Reduce wasted compute capacity
DAG Visualization
The DAG section displays the Spark execution plan as a Directed Acyclic Graph (DAG).
The DAG visualization helps you:
- Understand execution flow
- Analyze stage dependencies
- Review transformations and actions
- Identify expensive execution paths
For each node, you can view information such as:
- Operation Name
- Duration
- Output Rows
- Execution Relationships
Use the DAG view to perform detailed query and execution-plan analysis and to identify optimization opportunities.
Features
This page includes the following capabilities.
- Search and filter job information where applicable.
- Expand jobs to view detailed stage information.
- Switch between Jobs and Timeline views.
- Compare historical executions using Compare Runs.
- Analyze executor-specific metrics by enabling Show Individual Executors.
- Refresh the page to view the latest execution data.