Explanation
Why the waste happens and who it affects.
For short-running, bursty, or interactive workloads, Serverless can significantly reduce cost by billing only for execution time. However, Serverless is not universally available or compatible with all workload types and libraries. Organizations that exclusively rely on traditional clusters may be missing emerging opportunities to reduce spend and simplify operations by leveraging Serverless where appropriate.
Billing model
The pricing dimensions that drive this cost.
Databricks Serverless is billed based on:
- Execution time only
- Costs accrue only during active job runtime
- Databricks Units (DBUs): Based on usage, abstracted from specific VM types
- No direct infrastructure charges - Databricks manages the underlying infrastructure
Compared to traditional clusters, Serverless eliminates idle charges and provisioning overhead, and can reduce total compute spend for appropriate workloads.
How to detect
4 checks to find it in your estate.
- Query system tables to identify jobs or notebooks that run on traditional clusters despite short runtimes or high idle time
- Review whether Serverless Jobs Compute is enabled and available in the workspace
- Assess workload compatibility with Serverless requirements and feature support
- Analyze historical cluster usage for patterns that suggest better alignment with Serverless
How to fix
4 ways to remove the waste.
- Pilot Serverless for eligible workloads, such as short, periodic jobs or ad-hoc notebooks
- Use compute policies or templates to promote Serverless adoption where appropriate
- Retain traditional clusters for workloads with unsupported libraries or long-lived compute patterns
- Monitor performance and cost impact to guide incremental rollout and inform future defaults
Documentation
Vendor references for pricing and configuration.