# Lack of workload-specific cluster segmentation Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/lack-of-workload-specific-cluster-segmentation Running varied workload types (e.g., ETL pipelines, ML training, SQL dashboards) on the same cluster introduces inefficiencies. By: Nicole Boyd [Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) # Lack of Workload-Specific Cluster Segmentation The short version Running varied workload types (e.g., ETL pipelines, ML training, SQL dashboards) on the same cluster introduces inefficiencies. [Nicole Boyd](https://www.linkedin.com/in/nicolecboyd/) Managing Consultant - FinOps @ Capgemini Invent Databricks service [Databricks Compute](https://www.pointfive.co/efficiency-hub/cloud-services/databricks-compute) Category [Compute](https://www.pointfive.co/efficiency-hub/service-category/compute) Reference CER-0189 Type Inefficient Configuration ## Explanation Why the waste happens and who it affects. Each workload has different runtime characteristics, scaling needs, and performance sensitivities. When mixed together, resource contention can degrade job performance, increase cost, and obscure cost attribution. ETL jobs may overprovision memory, while lightweight SQL queries may trigger unnecessary cluster scale-ups. Job failures or retries may increase due to contention, and queued jobs can further inflate runtime costs. Without clear segmentation, teams lose the ability to tune environments for specific use cases or monitor workload-specific efficiency. ## Billing model The pricing dimensions that drive this cost. Databricks charges per-node-hour using Databricks Units (DBUs), with different rates based on cluster type and configuration. When disparate workloads share a single cluster - especially in all-purpose clusters - compute is inefficiently allocated, jobs contend for resources, and DBU consumption can spike due to overprovisioning, retry inflation, or queuing delays. ## How to detect 5 checks to find it in your estate. - Identify all-purpose or shared clusters that execute a wide range of job types - Look for clusters with inconsistent scaling or job runtimes - Review job metadata (e.g., task type, frequency, team owner) and compare to cluster setup - Examine cluster tags and naming conventions for clarity on intended usage - Check whether long-lived clusters are being used by multiple teams or pipelines ## How to fix 6 ways to remove the waste. - Define and enforce separate cluster types for distinct workload categories (e.g., SQL, ML, ETL) - Encourage the use of job clusters for short-lived, batch-oriented workloads to ensure clean isolation and efficient resource use - Use job clusters for single-purpose, short-lived jobs to ensure isolation and efficient spin-up - Apply strict cluster tagging and naming standards to reflect usage intent - Implement cluster policies that restrict configuration options based on workload class - Educate platform users on workload characteristics and recommend cluster segmentation best practices ## Documentation Vendor references for pricing and configuration. - [Cluster Policies docs.databricks.com](https://docs.databricks.com/aws/en/admin/clusters/policies) ## Related inefficiencies [Browse the library](https://www.pointfive.co/efficiency-hub) - Databricks Compute CER-0222 ### [Poorly Configured Autoscaling on Databricks Clusters](https://www.pointfive.co/efficiency-hub/inefficiencies/poorly-configured-autoscaling-on-databricks-clusters) Autoscaling is a core mechanism for aligning compute supply with workload demand, yet it's often underutilized or misconfigured. In older clusters or ad-hoc environments, autoscaling may be disabled by default or set with tight min/max... Compute - Databricks Compute CER-0195 ### [Overuse of Photon in Non-Production Workloads](https://www.pointfive.co/efficiency-hub/inefficiencies/overuse-of-photon-in-non-production-workloads) Photon is frequently enabled by default across Databricks workspaces, including for development, testing, and low-concurrency workloads. In these non-production contexts, job runtimes are typically shorter, SLAs are relaxed or nonexistent,... Compute - Databricks Compute CER-0523 ### [Idle Instances Retained in Databricks Instance Pools](https://www.pointfive.co/efficiency-hub/inefficiencies/idle-instances-retained-in-databricks-instance-pools) Databricks instance pools keep a set of ready cloud instances so classic clusters can start and scale faster. While an instance sits idle in a pool, Databricks charges no DBUs for it, but the cloud provider still bills the virtual machine.... Compute --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.