Explanation
Why the waste happens and who it affects.
This leads to idle compute, overprovisioning, and high baseline costs, especially when the clusters are always-on. Databricks SQL Serverless is optimized for bursty, interactive use cases with auto-scaling and pay-per-second pricing, making it better suited for this class of workloads. Failing to migrate to serverless for these patterns results in unnecessary cost without performance benefit.
Billing model
The pricing dimensions that drive this cost.
Serverless SQL compute is billed in DBUs based on the size of the SQL warehouse, for as long as the warehouse is running rather than only while queries execute. Idle time is still billed until Auto Stop shuts the warehouse down, but serverless allows a short Auto Stop setting (default 10 minutes, as low as 5 minutes in the UI or 1 minute via the API) and scales automatically based on demand. Continuing to use traditional job or all-purpose clusters for low-throughput SQL workloads results in persistent infrastructure costs that serverless options are purpose-built to minimize.
How to detect
4 checks to find it in your estate.
- Identify SQL warehouses (classic or pro) or clusters supporting dashboard or ad-hoc workloads with long idle periods between executions
- Review query duration and frequency - flag warehouses that are infrequently used or support low-throughput workloads
- Check whether teams are using dedicated clusters for interactive BI tools (e.g., Power BI, Tableau)
- Look for cost spikes associated with idle or underutilized SQL clusters versus expected user interaction volume
How to fix
4 ways to remove the waste.
- Migrate lightweight SQL workloads and dashboards to Databricks SQL Serverless
- Enable serverless for high-concurrency, low-compute scenarios where persistent compute isn't needed
- Set policies or guidelines to default to serverless for interactive workloads unless specific performance reasons require otherwise
- Educate data teams on the appropriate use cases for serverless versus provisioned clusters
Documentation
Vendor references for pricing and configuration.