# Missing Delta optimization features for high-volume tables Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/missing-delta-optimization-features-for-high-volume-tables In many Databricks environments, large Delta tables are created without enabling standard optimization features like liquid clustering (or, on older... By: Scott Shulman Updated: 2026-09-28 [Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) # Missing Delta Optimization Features for High-Volume Tables The short version In many Databricks environments, large Delta tables are created without enabling standard optimization features like liquid clustering (or, on older tables, partitioning and Z-Ordering). [Scott Shulman](https://www.linkedin.com/in/shulmanscott/) Automation and FinOps Evangelist @ Arrow Electronics Databricks service [Delta Lake](https://www.pointfive.co/efficiency-hub/cloud-services/delta-lake) Category [Storage](https://www.pointfive.co/efficiency-hub/service-category/storage) Reference CER-0177 Type Inefficient Architecture ## Explanation Why the waste happens and who it affects. Without these, queries scanning large datasets may read far more data than necessary, increasing execution time and compute usage. - Liquid clustering organizes data by clustering keys on commonly filtered columns to reduce scan scope, and keys can be changed without rewriting existing data. - Partitioning and Z-Ordering are older layout techniques; Databricks advises against partitioning tables under 1 TB, and neither can be combined with liquid clustering. - Delta Format enables additional optimizations like data skipping and compaction. Failing to use these features in high-volume tables often results in avoidable performance overhead and elevated spend, especially in environments with frequent exploratory queries or BI workloads. ## Billing model The pricing dimensions that drive this cost. Databricks charges are based on DBUs (Databricks Units) per hour, which correlate directly with compute resource use. Query performance heavily impacts DBU consumption. Inefficient data layout leads to longer scan times, increased cluster runtime, and higher costs. ## How to detect 5 checks to find it in your estate. - Large tables lacking liquid clustering on commonly filtered columns - Legacy tables without liquid clustering that lack Z-Ordering on high-selectivity columns (e.g., timestamps, IDs), or that are partitioned despite holding under 1 TB - Slow query performance tied to full-table scans - High DBU usage by queries reading large volumes of data unnecessarily - ETL pipelines writing to Delta tables without compaction or OPTIMIZE steps ## How to fix 5 ways to remove the waste. - Apply liquid clustering (CLUSTER BY) to Delta tables, using columns commonly filtered in queries, and avoid partitioning tables under 1 TB - On legacy tables that don't use liquid clustering, apply Z-Ordering on appropriate columns to improve data skipping efficiency - Use OPTIMIZE (or predictive optimization) to compact small files and improve query performance, and use VACUUM to remove data files no longer referenced by the table, which reduces storage cost but does not speed up queries - Standardize use of Delta Lake format in ETL pipelines - Enable predictive optimization so OPTIMIZE runs automatically on Unity Catalog managed tables, and automate periodic optimizations for other long-lived tables ## Documentation Vendor references for pricing and configuration. - [Optimize data file layout learn.microsoft.com](https://learn.microsoft.com/en-us/azure/databricks/tables/operations/optimize) - [Use liquid clustering for tables learn.microsoft.com](https://learn.microsoft.com/en-us/azure/databricks/tables/clustering) - [When to partition tables on Azure Databricks learn.microsoft.com](https://learn.microsoft.com/en-us/azure/databricks/tables/partitions) - [OPTIMIZE learn.microsoft.com](https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-optimize) - [Remove unused data files with vacuum learn.microsoft.com](https://learn.microsoft.com/en-us/azure/databricks/tables/operations/vacuum) ## Related inefficiencies [Browse the library](https://www.pointfive.co/efficiency-hub) - Delta Lake CER-0529 ### [Unvacuumed Delta Tables Retaining Unreferenced Data Files](https://www.pointfive.co/efficiency-hub/inefficiencies/unvacuumed-delta-tables-retaining-unreferenced-data-files) Delta Lake never modifies data files in place. Every UPDATE, DELETE, MERGE, overwrite and OPTIMIZE writes new files and marks the old ones as removed in the transaction log, but the old files stay in cloud object storage so earlier table... Storage - GCP GCS CER-0336 ### [Excessive API Request Cost Overhead from I/O-Intensive Workloads](https://www.pointfive.co/efficiency-hub/inefficiencies/excessive-api-request-costs-from-filesystem-style-access-on-gcs) Google Cloud Storage exposes a flat namespace through an HTTP API, but many workloads consume it through filesystem-style abstractions - FUSE mounts, Hadoop/Spark connectors, or analytic engines that enumerate prefixes to discover state.... Storage - Azure Blob Storage CER-0335 ### [Excessive Transaction Cost Overhead on Blob Storage from I/O-Intensive Workloads](https://www.pointfive.co/efficiency-hub/inefficiencies/excessive-transaction-costs-from-filesystem-style-access-patterns-on-blob-storage) Azure Blob Storage and ADLS Gen2 bill per transaction - every list, read, write, rename, and metadata operation is a separately metered API call. When organizations migrate workloads from on-premises Hadoop/HDFS environments or local... Storage --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.