Explanation
Why the waste happens and who it affects.
When used to run scheduled jobs or production workflows, they often stay idle between executions - leading to unnecessary infrastructure and DBU costs. Job clusters are designed for ephemeral, single-job execution and automatically terminate upon completion, reducing runtime and isolating workloads. Using interactive clusters for production jobs leads to cost inefficiencies and weaker workload boundaries.
Billing model
The pricing dimensions that drive this cost.
Databricks compute costs consist of:
- Databricks Units (DBUs): billed per second at a rate that depends on the compute type and instance; all-purpose (interactive) compute is billed at a higher DBU rate than jobs compute, so scheduled jobs on interactive clusters pay a premium on every DBU in addition to any idle time
- Cloud Infrastructure Charges: Passed through from the underlying cloud provider, billed per second or minute
Interactive clusters accrue charges continuously while running, regardless of workload activity. Job clusters are provisioned on demand and shut down automatically, incurring charges only for the job's runtime.
How to detect
5 checks to find it in your estate.
- Identify scheduled jobs assigned to interactive clusters
- Query system tables or logs to verify cluster assignment per job
- Review cluster uptime relative to job duration to assess idle overhead
- Evaluate whether workspace policies restrict interactive cluster use for jobs
- Confirm with engineering teams if job clusters can be safely adopted
How to fix
3 ways to remove the waste.
- Reassign scheduled jobs to ephemeral job clusters
- Apply workspace policies to enforce job cluster usage for scheduled workflows
- Educate users on the differences between cluster modes and their appropriate use cases
Documentation
Vendor references for pricing and configuration.