Databricks Compute

Databricks Compute gives you a tab-by-tab view of your Databricks environment — cluster health, job performance, resource utilization, and cost — so you can monitor and optimize your clusters without switching tools.

Databricks Compute has six tabs: Overview, Clusters, Job Studio, All Purpose Cluster, Job Runs, and DLT Pipelines.

Filters

Every tab includes a Data Source Filter, letting you switch between connected Databricks accounts or projects. Widget and table values reflect the time range selected in the Global Calendar filter unless noted otherwise.

Overview

The Overview tab summarizes cluster performance, resource utilization, and errors across your environment.

Widget	What It Shows
Cluster States	The count of clusters in each state — Pending, Running, Restarting, Resizing, Terminating, and Terminated — within the selected time range.
Databricks Users and Applications	The number of distinct users and applications active during the selected time range.
Average Core Usage Summary	Total Cores, Allocated Cores, and Used Cores across your environment.
Average Memory Utilization Summary	Total Memory, Allocated Memory, and Used Memory across your environment.
Databricks Top 10 Users	The top 10 users by number of clusters provisioned. Hover over a bar to see the count for that user.
Cluster Count by Instance Type	The distribution of clusters across instance types.
Active Clusters Over Time	The number of active clusters at each point in the selected time range.
Cluster Failure Over Time	Cluster failures over the selected time range, with the associated error code shown on hover. You can filter this graph by error code.
Top Cluster Errors	The most frequently occurring cluster errors, with occurrence count and error message.
DBU Consumed	Databricks Units consumed over the selected time range.
Average CPU Usage	CPU used per node, across all cluster types, over the selected time range.
Average Memory Used	Memory used per node, across all cluster types, over the selected time range.
Average Core Usage	Available, allocated, and used cores over the selected time range, as separate trend lines.
Average Memory Utilization	Available, allocated, and used memory over the selected time range, as separate trend lines.
Core Wastage Over Time	Unused CPU cores over the selected time range — useful for spotting idle capacity.

Use Cluster States and Cluster Failure Over Time together to spot clusters that need troubleshooting. Use DBU Consumed to monitor usage trends before they affect cost. Use Top Cluster Errors to prioritize which recurring issues to resolve first.

Enhanced filtering

The search filter on this tab supports a broader set of filterable columns — including cluster status, source, duration, and user — with a few refinements:

Columns already visible in the table are hidden from the filter dropdown, to keep the list manageable.
Filter options adapt to your current view, so only relevant choices are shown.
Filters persist as you navigate, so you don't need to reapply them.
The equals (=) operator is the primary filter interaction.

Clusters

The Clusters tab lists every cluster with its cost, status, and configuration.

Column	Description
Cluster Name	The cluster's name. This column stays visible when you scroll horizontally. Selecting it opens the Job Studio page.
Cluster ID	A system-generated identifier. Selecting it opens the cluster's past runs.
Status	Running, Terminated, Pending, or Resizing.
Duration	Total active time, from start to end (or to now, if still active).
Total DBU Consumed	Total Databricks Units consumed by the cluster.
Actual Databricks Cost	Total cost from Databricks services for the workload.
Actual Cloud Total Cost	Combined cost of all cloud resources consumed.
Actual Cloud VM Cost	Cost specifically from virtual machine usage.
Recommended Cloud VM Cost	Estimated cost if a more optimal VM configuration were used.
Recommended Instance Type	A suggested VM instance type for better cost-efficiency or performance.
Start Time / End Time	When the cluster started and terminated. End Time is empty if the cluster is still running.
Cluster Source	How the cluster was created — Job, API, UI, or Pipeline.
User	The user who initiated or is running the cluster.
Termination Type / Termination Code	How and why the cluster terminated — for example, Success, Client Error, or User Request.
Diagnostic Reason	Detailed diagnostic information about termination or errors.
Spark Version	The Apache Spark version running on the cluster.
Worker Node Type / Driver Node Type	The instance types used for worker and driver nodes.

Cluster details

Select a cluster name to open its details page, showing a Past Runs chart (DBU count and cost by date) and a Past Job Runs Details table.

Column	Description
Creation Time	When the cluster was created.
State	The cluster's or job's current state.
DBU Consumed	Databricks Units consumed.
Start Time / Termination Time	When the job execution began and completed.
Executor Config	The executor settings and specifications.
Number of Workers / Min Workers / Max Workers	Worker node counts allocated to the job.
Executor Memory	Memory capacity allocated to the cluster.
Duration	Total execution time.
Balanced Recommendation / Cost Recommendation / Runtime Recommendation	Sizing recommendations for each optimization goal.
State Message	The message associated with the cluster's state.
Username	The user associated with the run.

A single job run can be reviewed on this page even without prior runs to compare against.

Job Studio

Job Studio gives you a tabular view of every Databricks job, with filtering to track, monitor, and manage them.

Known Limitation All cost data in ADOC is shown in US Dollars (USD); currency conversion isn't supported. If your Databricks account reports costs in another currency, ADOC displays the same numerical value labeled as USD. This applies to both actual and estimated cost charts.

A chart at the top of the page shows job counts over time, broken down by status: Canceled, Failed, and Success. The table below lists every job matching your filters.

Column	Description
Job Name	The job's name.
Cluster ID	The cluster running the job.
Job Status	Success, Failed, or Canceled.
Actual Databricks Cost / Estimate Databricks Cost	The actual and estimated cost from Databricks resources.
Estimate Vendor Cost	Estimated cost from third-party vendor resources.
Total Job Cost	Combined Databricks and vendor cost.
Start Time / End Time / Duration	When the job ran and how long it took.
Vendor Storage Cost / Vendor Virtual Machines Cost / Vendor Virtual Network Cost / Vendor Bandwidth Cost	Cost breakdown by vendor resource category.
Run Page URL	A link to the job's run page for logs, metrics, and performance data.
Cluster State	The state of the cluster running the job.
Creator User	Who created the job.
Trigger	PERIODIC (scheduled) or ONE-TIME (manual).
Runtime Engine	Photon or Standard.
Job ID / Run ID	Identifiers for the job and this specific run.

Job Studio also provides:

Preset views for Top 20 Expensive Jobs and Long Running Jobs, to surface resource-intensive tasks quickly.
Download to export job data for offline analysis or sharing.
Combinable filters by status, creator, runtime engine, and more.

Job details page

Select a job to see driver and executor performance, trends, and resource usage in depth.

Summary

Field	Description
Actual Databricks Cost	Cost from Databricks resources for this job, based on DBU and platform resource consumption.
Actual Vendor Cost	Cost from third-party vendor resources used alongside the job.
Total Cost	Combined Databricks and vendor cost.
Cluster ID	The cluster the job ran on.

Vendor cost breakdown

Field	Description
Virtual Machines Cost	Cost of vendor-provided virtual machines used for compute or supporting services.
Storage Cost	Cost of external data storage — intermediate files, logs, or outputs.
Virtual Network Cost	Cost of vendor-managed network infrastructure.
Bandwidth Cost	Cost of data transferred between systems or across network boundaries.

Job run details availability: Detailed run metrics require the Databricks initialization script to be enabled. Jobs run before onboarding or before the script was enabled won't have detailed metrics available. Configure the init script at onboarding time to ensure metrics are captured going forward.

Node size recommendations

Node size recommendations suggest how to configure Spark executor nodes based on cost, runtime, and workload characteristics.

Static clusters (fixed worker count) get recommendations for optimal core count, memory per executor, and number of workers.
Auto-scale clusters get recommended minimum and maximum worker counts, estimated completion time, and cost for each configuration.

Recommendations are based on:

Metric	What It Drives
CPU Utilization	High usage suggests more cores per executor; low usage suggests fewer.
Memory Utilization	High usage suggests more memory per executor; low usage suggests reducing allocation.
Shuffle Operations	Shuffle fetch wait time and remote bytes read factor into whether additional executors are needed.

Recommendations aren't available for single-node clusters, jobs without Spark stages, failed or cancelled jobs, or all-purpose clusters (which auto-scale dynamically, making static recommendations less useful).

Driver and executor summary

Field	Description
Name	The driver instance identifier.
User	The account that initiated the driver.
Duration	How long the driver was active.
Max Heap Used	Peak heap memory consumed by the driver.
Instance Type	The VM or hardware configuration used for the driver.

Field	Description
Cores	CPU cores allocated to the driver.
Memory Available	Total memory allocated to the driver.
Jobs / Stages	The number of jobs and stages processed by the executors.
Max Used Memory	Peak memory usage by the executors.
Instance Type / Cores per Instance / Memory Available	The executor instance configuration.
Total Instances	The number of executor instances used.

Executor node recommendation

Recommendations are provided for three optimization types: Cost-Optimized, Runtime-Optimized, and Balanced, for both auto-scale and static cluster configurations, and across different instance types. Each recommendation includes estimated completion time, worker count (or min/max range for auto-scale), and estimated vendor cost, so you can compare trade-offs before choosing a configuration.

Trends

Shows Executor Memory, Executor Cores, and Input Bytes Read over time. Use Compare Runs to compare these trends across different job runs.

Limits

Analyzes scalability constraints using:

Wall Clock Time — Driver, Executor, and Total wall clock time.
Ideal Times — Critical Path (minimum possible completion time), Ideal Application Time, and Actual Runtime.
OOCH (One Core Compute Hour) — Available versus wasted compute hours, broken down by executor and driver.

Metrics

Metric	Description
Storage Memory	On-heap and off-heap memory allocated, used, and available.
Schedule Information	Active tasks and thread pool size over time.
Bytes Read/Written	Total data read and written by the job.
File System Bytes Read/Written	Bytes read and written directly to and from the filesystem.
Shuffle Information	Bytes written and read during shuffle operations, from local and remote sources.
Spark JVM GC and CPU Time	JVM garbage collection time and CPU time — high GC time can indicate inefficient memory usage.
Records Read/Written	Records read and written during execution.

Spark details aggregate metrics

Metric	Description
Task Duration	Total time spent by the task from creation.
JVM GC Time	Time spent in garbage collection while the task was in progress.
Executor CPU Time	CPU time spent by the executor running the task, including shuffle data fetch.
Executor Deserialize CPU Time / Time	CPU time and elapsed time spent deserializing the task.
Executor Runtime	Total time spent by the executor core running the task.
Peak Execution Memory	Maximum execution memory used by the task.
Input Bytes Read / Output Bytes Written	Bytes read and written by the task via the respective APIs.
Disk Bytes Spilled / Memory Bytes Spilled	Bytes spilled to disk during the task.
Result Size	Bytes sent back to the driver.
Result Serialization Time	Time spent serializing the task result.
Shuffle Read Bytes / Fetch Wait Time / Local Blocks / Records Read / Remote Blocks	Shuffle read details for the task.
Shuffle Write Bytes Written / Records Written / Time	Shuffle write details for the task.

Spark SQL executions

Field	Description
Execution ID	A unique identifier for the SQL execution.
Description	The query or operation being executed.
Start Time / End Time / Duration	When the execution ran and how long it took.
State	Running, Completed, or another status.
More Details	A link to deeper insight into the query's performance and execution plan.

Stages

Available as a List (tabular breakdown of tasks and performance per stage) or Timeline (each stage shown as a horizontal bar).

Driver and executor stats

CPU Usage, Memory Usage, Heap Usage, and Core Wastage, each shown for both driver and executors, so you can identify underutilization, memory inefficiency, or over-provisioning.

All Purpose Cluster

This tab breaks down cost for all-purpose clusters, individually.

A left-side panel lists every cluster with its total cost, searchable by name. Selecting a cluster updates the chart and table.
A bar chart shows Total Cost over time for the selected cluster, with per-date detail on hover.
A breakdown table shows Databricks Cost, Vendor Cost, and Total Cost by date, sortable by any column.

Use Download to export the cost data for the selected cluster.

Job Runs

This tab lists completed and ongoing jobs, filterable by:

Cluster Type — job clusters or all-purpose clusters.
Status — Success, Failed, Canceled, or Running.
Owner — the user who initiated the job.

Column	Description
Cluster Name / Cluster ID / Cluster Type	The cluster associated with the job run.
Job Name / Job ID	The job's name and unique identifier.
Status	SUCCESS, FAILED, or CANCELED.
Duration	Time taken to complete the job.
DBU Consumed	Databricks Units consumed during the run.
Estimated Databricks Cost / Estimated Vendor Cost	Estimated cost from Databricks and the cloud vendor.
Start Time / End Time	When the job ran.
Executor Heap Used % / CPU Used %	Resource utilization by the executor during execution.
Executor Memory	Total memory allocated to the executor.
Diagnostics	Errors or workload details for the job.
Owner	Who owns or initiated the job.
App Id / App Name	The application or job instance identifier and label.

DLT Pipelines

This tab manages and monitors Delta Live Tables pipelines, filterable by:

Current State — Idle, Running, Failed, and similar.
Owner — the user or team managing the pipeline.

Column	Description
Name	The pipeline's name.
Current State	The pipeline's operational state.
Owner	Who owns or manages the pipeline.
Pipeline Execution	The environment the pipeline runs in — for example, Development.
Pipeline Mode	How the pipeline runs — for example, Triggered.
Total Runs	Total execution count.
Last Job Run	A link to the most recent execution.
Last Run State / Duration / Start Time	The result, duration, and start time of the most recent run.

Select a pipeline name to open its Pipeline Run Details panel.

Field	Description
Pipeline Name	The pipeline being executed.
Cluster ID	The cluster associated with the run.
State	FAILED, SUCCESSFUL, or similar.
Cause	The reason for the run's state — for example, JOB_TASK.
Start Time / End Time	When the run started and ended.
Is Validate Only	Whether the run was validation-only.
Is Full Refresh	Whether the run performed a full refresh.
Execution	Execution details.
Databricks Cost / Cloud Vendor Cost / Total Cost	The cost breakdown for the run.

What's next

Databricks Cost - See cost trends broken down by cluster type and vendor service.
Databricks Query Studio – Investigate individual query performance and cost.

Last updated on

Was this page helpful?