Spark Jobs
The Spark Jobs page provides a Sankey diagram with jobs by users and by queue. The page also provides detailed information about Spark jobs categorized by name of the job, users running the job, queue in which the job is run, status of the job, and other criteria.
Go to Spark > Jobs to view the Spark Jobs page.
The default time is Last 24 hrs. To change the timeline, click the down arrow in the time selection menu.
The default grouping is Queue. To change the grouping, perform the following:
Click Group-By drop-down menu in the top right corner of the page, the drop-down menu is displayed.
Select the type of grouping you want to display from the following list:
- Queue
- User
- Name
- Status
The details by selected type is grouped in the table.
(Optional)To ungroup the queries, select Ungroup.
The following tables provides description of the metrics displayed for each job group:
| Metric | Description |
|---|---|
| Group Name | The name of the group. |
| Job Count | The total number of jobs associated by a particular group. |
| Failed Jobs | The total number of failed jobs associated by a particular group. |
| Running Jobs | The total number of running jobs associated by a particular group. |
| Avg Duration/Job | The average time taken to execute a job by a particular group. |
| Total Duration | The total time taken to execute jobs by a particular group. |
| Avg Mem/Job | The average amount of memory consumed by a particular job group. |
| Total Mem | The total memory consumed by a particular job group. |
| Avg VCore/Job | The average number of VCores used by a particular job group. |
| Total VCore | The total number of VCores used by a jobs by a particular group. |
Click on the group row to view the details of the jobs in that group. The following tables provides description of the metrics displayed for each job:
| Metric | Description |
|---|---|
| App ID | The application ID of the job. To display the concurrency of the job, click ... icon and click View Concurrency. |
| Attempt ID | The number of attempts made by the job. |
| User | The name of the user who executed the job. |
| Name | The name of the spark job. To copy the name of the job, hover over the name you want to copy and click the |
| Final Status | The final state of the Spark job. The following are the status: Failed, Finishing, Killed, Running, Succeeded. Click the log icon to view the Logs for a specific job. |
| Dynamic Executer | Displays the execution condition for Spark Jobs such as True or False.
|
| Queue | The name of the queue that the job belongs to. |
| Duration | The total time taken to execute the job. |
| Started Time | The starting time of the job |
| Finished Time | The job's completion time. |
| Memory | The amount of memory consumed in executing the job. |
| VCores | The number of VCores used in job processing. |
Click the App ID to view more details about the job in the Spark Job Details page.
Filtering Spark Jobs
To filter the data you want to view in the table, click the filter drop-down menu and select the filters. The following are the drop-down menu options:
- User
- Name
- Final Status
- Queue
You can search for the required filter using the Search box provided in the drop-down menu. Click Reset to clear the applied filter.
Searching Spark Jobs
You can use the search box to search for a job. For example, the name of the parameter visible in the UI is User. However, internally, the parameter name is effective_user. Hence, if you want to search for queries with User = Mark, you need to type effective_user:Mark.
Click Reset to clear the applied filter.