Spark Job Details

The Spark Job Details page contains the following panels:

• Summary Panel

• Job Trends

• Configurations

• Spark Stages

• Time Series Information

• Reports

• Application Logs

Note

If you cannot view the Spark Jobs details page, you can view the instructions given on the page and follow them so that you can view the Spark job details.

Summary Panel

The top panel displays the following information:

Job Trends

The Job Trends panel displays a chart showing the pattern of jobs running at a particular time.

The following table provides description of the factors displayed in the Job Trends chart:

You can choose the metrics displayed in the Job Trends chart. To choose the metrics, perform the following:

1. Click the Comparison metrics drop-down menu in the chart. The list of the metrics is displayed.

2. Select the metrics you choose to view. The data respective to the selected metrics is displayed in the chart.

3. (Optional) To add metrics, click and select the metrics from the list.

4. (Optional) To remove the selected metrics, click corresponding to the metrics you want to remove.

Switching Job Trends View

You can switch between bar chart view and line chart view.

• Click , to view the Job Trends as a bar chart.

• Click to view the Job Trends as a line chart.

Configuration Difference

Click Compare Runs to compare different configuration of the job. Select the runs that you want to compare from the drop-down list. You can choose from up to 10 previous runs of the job. Apart from the previous 10 runs, you can also choose to compare an execution of the job with any other execution. You must click the Enter App ID button. You must enter the application ID, followed by an underscore and then the original attempt in the following format.

app.id_originalattempt

Configurations

With Configurations, you can view the Job Configurations and Anomalous hosts.

Job Configurations

Job Configurations displays the Current Value and the Recommended Value for the following parameters:

Node Issues

This section displays the unhealthy nodes of this Spark application. Blacklisted nodes, nodes on which Kernel errors are reported, nodes whose disk utilization is close to being full are some examples which make a node unhealthy. This section displays a table with two columns. The HEALTH REPORT column displays the reasons as to why a node is marked unhealthy and the NODE LIST column displays the list of nodes which are unhealthy.

Anomalies

Anomalies board displays system metrics for the host which is used by the Spark job within the duration of that job. The host can be impacted by the usage of CPU, Memory, Network, or Disk.

With Anomalous data, you can monitor the host performance and make predictions on memory, CPU, Network I/O, and disk usage.

To view more details about Anomalous hosts, click the host link in the Anomalies tab.

You can detect anomalies based on the following metrics.

Error Details

The error information for the job that failed to complete is displayed. This panel is displayed for a job that has a final status of FAILED.

To copy the error details, perform the following:

1. Click the copy icon. A confirmation dialog-box appears.

2. Click Ok. The error details are copied to the clipboard.

Concurrent Running Apps

This table displays the list of applications which were running parallelly with the selected application. You can view the following details in this section.

Spark Jobs

This section displays the complete hierarchy of Spark applications. A Spark application consists of Jobs. A job can further have many stages and each stage can have tasks. In this chart, you can view the jobs, stages and tasks of the application.

Each job has a job ID. You can expand a job to view its stages.

This section has the following columns.

You can choose to group or ungroup the data by Job ID. To group, you must select the Group By job Id check box. Also, when you ungroup you can choose to filter out inefficient stages.

You can click more details button to view the tasks of a Stage.

You can click the DAG tab to view the execution flow only for the specific task.

Spark Stages

Stages are units in which a job is divided into small tasks. You can view Spark Stages in the form of a List or a Timeline. Click More Details to view the details of a particular stage.

Timeline

The timeframe in which tasks in the stage executed. The timeline also includes the total driver execution time. You can sort the timeline of these tasks by Duration and Start Time other than the default view. To sort, perform the following:

1. Click SortBy in the chart. The drop-down menu is displayed.

2. Select the type of sort you want to apply. The data corresponding to the selected sort is displayed.

3. (Optional) Select None, to remove the applied sort.

Viewing Timeseries Information

By default, the Timeseries Information charts display information on all the stages. To view Timeseries Information of an individual stage, perform the following:

• Select the job stage from the list. The Timeseries Information for the selected stage is displayed.

• To view information of all the stages, select the Show all Stages checkbox. The Timeseries Information for all the stage is displayed.

Timeseries Information

Timeseries information displays timeseries metrics of the application you are currently viewing. Within the time duration, you can see the time spent by the drivers, denoted by a red box. The drivers help in running Spark applications as sets of processes on a cluster.

Note You can see the name of the application you are currently viewing, above the user name in the top panel.

Time Series Information Charts

The following table provides description of the Timeseries Information charts:

Info By moving the slider, you can specify the time range for driver execution. The data in the timeseries charts is displayed based on the range of time selected.

By default all the charts displays the aggregated executors. To view individual executors, perform the following:

1. Click Show Individual Executors. A drop-down menu of the executors is displayed.

2. Select the executor from the drop-down menu. The data corresponding to the selected executor is displayed in the chart.

3. To view the aggregated executors, click Show Aggregated Executors.

Reports

The Reports tab has the following tabs:

Efficiency Statistics

Driver versus executor time spent determines how well the Spark program has been written and if the right amount of parallelism is achieved.

The following tables provides the description of the charts in the Efficiency Statistics panel:

One Core Compute Hour (OCCH) is derived from one core running for one hour in the executor. The following metrics are displayed in the chart.

• OCCH wasted by driver.

• OCCH wasted by executor.

• Total OCCH wasted.

• Total OCCH available.

Simulation

The Simulation chart displays the Estimated Wall Clock Time and Cluster Utilization (%) for the number of executors used. You can determines what should be the ideal number of executors on the Spark program and what would be the effect of such changes to the number of executors on the overall time and utilization.

YARN Diagnostics

The YARN Diagnostics displays the details of the YARN application that was running in that duration executed by the user. The following table provides description of the table in the YARN Diagnostics tab:

To view more details, click on the row. The following table provides description of the table displayed:

Aggregate Metrics

The Aggregated Metrics tab displays the aggregated usage of different metrics in that application. The following table provides description of the metrics in the Aggregated Metrics tab:

Core Usage By Locality

The panel provides the information for the core usage by locality processed by Spark application. The following metrics is displayed in the charts:

• Process Local: The tasks in this locality are run within the same process as the source data

• Node Local: The tasks in this locality are run on the same machine as the source data.

• Wastage

Gluten SQL/DataFrame

When Gluten acceleration is enabled for a Spark job, Pulse provides detailed visibility into how query execution is handled by Gluten.

This section helps you understand:

• Which parts of the Spark physical plan were executed using Gluten

• Which operators fell back to standard Spark execution

• Why fallback occurred for specific plan nodes

Key Benefits

Using the Gluten SQL/DataFrame view, users can:

• Validate whether Gluten acceleration is applied effectively

• Identify fallback-heavy queries

• Understand operator-level execution behavior

• Improve performance tuning for Spark workloads

Gluten Nodes and Fallback Nodes

For each SQL/DataFrame query, Pulse displays:

• Num Gluten Nodes: The number of physical plan operators successfully accelerated by Gluten.

• Num Fallback Nodes: The number of operators that could not be executed using Gluten and were executed using Spark’s default engine instead.

Details

Click on a query to view the following:

Fallback Summary

• The Fallback Summary explains, at the plan-node level, which operators were not supported by Gluten.

• For each entry, Pulse shows the node ID, operator type (Scan, Project, Exchange, etc.), and a brief reason for fallback, such as unsupported file formats, Spark functions, backend validation failures, or unsupported operators.

• This helps identify parts of the query that did not benefit from acceleration.

Physical Plan View

• Pulse displays the full Spark Physical Plan with Gluten execution markers.

• It highlights the operator hierarchy, Gluten-accelerated nodes (ExecTransformer), fallback nodes executed in Spark row mode, and columnar-to-row transitions, providing end-to-end visibility into query execution.

DAG

The DAG chart displays the exact flow of how each task has been executed. The DAG flow displayed on this section represents the entire application. You can view the flow in which all the tasks in the application are executed.