# Inefficient query design in Databricks SQL and Spark jobs Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/inefficient-query-design-in-databricks-sql-and-spark-jobs Many Spark and SQL workloads in Databricks suffer from micro-optimization issues - such as unfiltered joins, unnecessary shuffles, missing broadcast... By: Nicole Boyd Updated: 2026-09-28 [Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) # Inefficient Query Design in Databricks SQL and Spark Jobs The short version Many Spark and SQL workloads in Databricks suffer from micro-optimization issues - such as unfiltered joins, unnecessary shuffles, missing broadcast joins, and repeated scans of uncached data. [Nicole Boyd](https://www.linkedin.com/in/nicolecboyd/) Managing Consultant - FinOps @ Capgemini Invent Databricks service [Databricks SQL](https://www.pointfive.co/efficiency-hub/cloud-services/databricks-sql) Category [Compute](https://www.pointfive.co/efficiency-hub/service-category/compute) Reference CER-0220 Type Inefficient Configuration ## Explanation Why the waste happens and who it affects. These problems increase compute time and resource utilization, especially in exploratory or development environments. Disabling Adaptive Query Execution (AQE) can further exacerbate inefficiencies. Optimizing queries reduces DBU costs, improves cluster performance, and enhances user experience. ## Billing model The pricing dimensions that drive this cost. Databricks charges by Databricks Units (DBUs), which are billed per-second based on the compute resources used. Inefficient query design leads to longer execution times, increased memory and shuffle usage, and higher DBU consumption without proportional business value. ## How to detect 5 checks to find it in your estate. - Identify queries with long execution times and large shuffle stages - Review query plans for missing filters or non-optimal joins (e.g., lack of broadcast joins) - Check for repeated reads from large datasets without use of caching - Check memory and storage metrics for signs of inefficient caching (e.g., low cache hit rate despite frequent reads) - Evaluate whether Adaptive Query Execution (AQE) is enabled - it is on by default in recent Spark versions, but may be disabled in older clusters or overridden in session or job-level configurations. ## How to fix 7 ways to remove the waste. - Enable Adaptive Query Execution to improve join strategies and reduce shuffle - Use broadcast joins for small lookup tables where applicable - Apply filtering and predicate pushdown early in the query - Cache frequently accessed tables or intermediate query results - Refactor inefficient joins, subqueries, or nested aggregations - Promote a culture of reviewing physical query plans during development - Use liquid clustering (recommended for new tables) to improve scan efficiency for large tables, or partition pruning and Z-ordering on existing tables that don't use liquid clustering ## Documentation Vendor references for pricing and configuration. - [Work with joins on Databricks docs.databricks.com](https://docs.databricks.com/aws/en/transform/join) - [Optimize performance with caching on Databricks docs.databricks.com](https://docs.databricks.com/aws/en/optimizations/disk-cache) - [Performance Tuning - Spark Documentation spark.apache.org](https://spark.apache.org/docs/latest/sql-performance-tuning.html#adaptive-query-execution) - [Adaptive query execution docs.databricks.com](https://docs.databricks.com/aws/en/optimizations/aqe) ## Related inefficiencies [Browse the library](https://www.pointfive.co/efficiency-hub) - Databricks SQL CER-0225 ### [Underuse of Serverless for Short or Interactive Workloads](https://www.pointfive.co/efficiency-hub/inefficiencies/underuse-of-serverless-for-short-or-interactive-workloads) Many organizations continue running short-lived or low-intensity SQL workloads - such as dashboards, exploratory queries, and BI tool integrations - on traditional clusters. This leads to idle compute, overprovisioning, and high baseline... Compute - Databricks SQL CER-0522 ### [Long or Disabled Auto Stop on Databricks SQL Warehouses](https://www.pointfive.co/efficiency-hub/inefficiencies/long-or-disabled-auto-stop-on-databricks-sql-warehouses) A Databricks SQL warehouse keeps running after its last query until its Auto Stop timer expires, and Databricks states that idle SQL warehouses continue to accumulate DBU and cloud instance charges until they are stopped. For typical use... Compute - Databricks SQL CER-0521 ### [Oversized Databricks SQL Warehouses](https://www.pointfive.co/efficiency-hub/inefficiencies/oversized-databricks-sql-warehouses) A Databricks SQL warehouse's cluster size sets the driver instance and the number of workers in each of its clusters, from 2X-Small with one worker up to 5X-Large with 512, doubling at each step. The warehouse bills for that size for every... Compute --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.