# Poorly configured autoscaling on Databricks clusters Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/poorly-configured-autoscaling-on-databricks-clusters Autoscaling is a core mechanism for aligning compute supply with workload demand, yet it's often underutilized or misconfigured. By: Nicole Boyd Updated: 2026-09-28 [Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) # Poorly Configured Autoscaling on Databricks Clusters The short version Autoscaling is a core mechanism for aligning compute supply with workload demand, yet it's often underutilized or misconfigured. [Nicole Boyd](https://www.linkedin.com/in/nicolecboyd/) Managing Consultant - FinOps @ Capgemini Invent Databricks service [Databricks Compute](https://www.pointfive.co/efficiency-hub/cloud-services/databricks-compute) Category [Compute](https://www.pointfive.co/efficiency-hub/service-category/compute) Reference CER-0222 Type Inefficient Configuration ## Explanation Why the waste happens and who it affects. In older clusters or ad-hoc environments, autoscaling may be disabled by default or set with tight min/max worker limits that prevent scaling. This can lead to persistent overprovisioning (and wasted cost during idle periods) or underperformance due to insufficient parallelism and job queuing. Poor autoscaling settings are especially common in manually created all-purpose clusters, where idle resources often go unnoticed. Overly wide autoscaling ranges can also introduce instability: Databricks may rapidly scale up to the upper limit if demand briefly spikes, leading to cost spikes or degraded performance. Understanding workload characteristics is key to tuning autoscaling appropriately. ## Billing model The pricing dimensions that drive this cost. Databricks charges based on Databricks Units (DBUs) per node per hour. When autoscaling is misconfigured - such as by fixing the worker count (Min \\= Max), setting narrow ranges, or disabling it entirely - clusters may remain overprovisioned during idle periods or underprovisioned during peak demand, resulting in inefficient compute spend or job failures. ## How to detect 7 checks to find it in your estate. - Identify clusters with identical min and max worker values (fixed size) - Review clusters with low task parallelism or frequent job queuing - Check for long idle durations on high-capacity clusters - Analyze job failures caused by memory or executor limits - Use the Spark UI, Databricks compute metrics, or cluster event logs to monitor how often clusters scale up or down (Ganglia is not available on Databricks Runtime 13 and above) - Look for consistent underuse of allocated workers during job execution - Review logs or metrics for unexpected or excessive scale-up behavior ## How to fix 5 ways to remove the waste. - Use autoscaling for variable workloads, but avoid overly wide min/max ranges that allow clusters to over-expand. Databricks may aggressively scale up if limits are too high, leading to cost spikes and instability. - For predictable, recurring jobs with stable compute requirements, consider using fixed-size clusters to avoid the cost and time of scaling transitions. - Tune autoscaling thresholds based on real workload behavior. Start narrow and adjust iteratively, based on runtime performance and cluster utilization. - Establish cluster policies to enforce sensible auto scaling defaults or require justification for disabling autoscaling entirely. - Regularly review cluster usage patterns to refine scaling decisions. While some tools can automate this, most teams can start with basic monitoring of scaling events and job runtimes. ## Documentation Vendor references for pricing and configuration. - [Compute configuration reference docs.databricks.com](https://docs.databricks.com/aws/en/compute/configure) - [Classic compute configuration best practices docs.databricks.com](https://docs.databricks.com/aws/en/compute/cluster-config-best-practices) - [Create and manage compute policies docs.databricks.com](https://docs.databricks.com/aws/en/admin/clusters/policies) - [Databricks Pricing databricks.com](https://www.databricks.com/product/pricing) ## Related inefficiencies [Browse the library](https://www.pointfive.co/efficiency-hub) - Databricks Compute CER-0189 ### [Lack of Workload-Specific Cluster Segmentation](https://www.pointfive.co/efficiency-hub/inefficiencies/lack-of-workload-specific-cluster-segmentation) Running varied workload types (e.g., ETL pipelines, ML training, SQL dashboards) on the same cluster introduces inefficiencies. Each workload has different runtime characteristics, scaling needs, and performance sensitivities. When mixed... Compute - Databricks Compute CER-0195 ### [Overuse of Photon in Non-Production Workloads](https://www.pointfive.co/efficiency-hub/inefficiencies/overuse-of-photon-in-non-production-workloads) Photon is frequently enabled by default across Databricks workspaces, including for development, testing, and low-concurrency workloads. In these non-production contexts, job runtimes are typically shorter, SLAs are relaxed or nonexistent,... Compute - Databricks Compute CER-0523 ### [Idle Instances Retained in Databricks Instance Pools](https://www.pointfive.co/efficiency-hub/inefficiencies/idle-instances-retained-in-databricks-instance-pools) Databricks instance pools keep a set of ready cloud instances so classic clusters can start and scale faster. While an instance sits idle in a pool, Databricks charges no DBUs for it, but the cloud provider still bills the virtual machine.... Compute --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.