Explanation
Why the waste happens and who it affects.
However, Photon only accelerates workloads that use compatible operations and data patterns. If a workload includes unsupported functions, unoptimized joins, or falls back to interpreted execution, Photon may be silently bypassed - even on a Photon-enabled cluster. In this case, Photon instances on classic compute still consume DBUs at a higher rate than non-Photon instances while delivering no meaningful speed or efficiency gain. This inefficiency typically arises when teams enable Photon globally without validating workload compatibility or updating their pipelines to follow Photon best practices. The result is higher costs with no corresponding benefit - a classic case of configuration drift outpacing optimization discipline.
Billing model
The pricing dimensions that drive this cost.
Azure Databricks compute charges are based on Databricks Units (DBUs).
- On classic all-purpose, jobs, and pipeline compute, Photon instance types consume DBUs at a higher rate than the same instance type without Photon (Photon is enabled by default there); SQL warehouses and serverless compute always run Photon, with no separate Photon option
- If workloads are not optimized for Photon, users may incur higher costs without realizing any performance benefits
How to detect
4 checks to find it in your estate.
- Analyze job execution plans to determine whether Photon is being used end-to-end
- Review workloads that run on Photon-enabled clusters but show no runtime improvement over standard execution
- Identify SQL operations or UDFs that are unsupported by Photon
- Check for UDFs, RDD or Dataset API calls, stateful streaming, or other unsupported operations that force a fallback to the Spark runtime
How to fix
5 ways to remove the waste.
- Ensure that Photon is only enabled for workloads structured to benefit from vectorized execution
- Replace UDFs and RDD or Dataset API usage with built-in SQL or DataFrame functions that Photon can execute natively
- Use the Spark UI (classic compute) or the query profile (SQL warehouses and serverless) to check which operators and how much task time run in Photon
- Monitor DBU consumption alongside job duration to track whether Photon is delivering net cost/performance gains
- Collaborate with data engineering teams to continuously tune high-volume pipelines for Photon compatibility
Documentation
Vendor references for pricing and configuration.
