Explanation
Why the waste happens and who it affects.
Each cluster must initialize independently, adding latency and cost. This results in inefficient resource usage, especially for workflows that could reuse the same cluster across tasks. Consolidating tasks onto a single job cluster where feasible reduces start-up time and avoids duplicative compute charges.
Billing model
The pricing dimensions that drive this cost.
Databricks is billed based on Databricks Units (DBUs) consumed by the compute resources provisioned. Each job cluster accrues DBU charges while running, and additional cost arises from repeated cluster start-up times and redundant resource allocation.
How to detect
4 checks to find it in your estate.
- Review whether multiple clusters are being provisioned within a single workflow for tasks with similar compute requirements
- Assess cluster start-up overhead across tasks to determine if repeated initialization is adding latency and cost
- Examine job execution logs to confirm whether resource utilization across clusters is consistent and could be consolidated
- Evaluate whether the workflow is creating clusters unnecessarily instead of leveraging a single reusable job cluster
How to fix
3 ways to remove the waste.
- Configure a shared job cluster to run multiple tasks within the same workflow when compute requirements are similar
- Leverage cluster reuse settings to reduce start-up overhead and improve efficiency
- Validate that consolidation does not impact workload performance or isolation requirements before implementing
Documentation
Vendor references for pricing and configuration.