Explanation
Why the waste happens and who it affects.
These formats are row-based and verbose, requiring Athena to scan and parse the full content even when only a few fields are queried. Without columnar formats, partitioning, or metadata-aware table formats, queries become inefficient and expensive, especially in high-volume environments.
Billing model
The pricing dimensions that drive this cost.
Pay-per-scan - Athena charges based on the amount of data scanned per query, not the size of the result set. This makes storage format, partitioning, and file layout critical cost drivers.
How to detect
5 checks to find it in your estate.
- Review whether data stored in S3 is in raw formats like JSON or CSV
- Check if files are written at high frequency, resulting in large numbers of small files
- Evaluate whether data is partitioned by commonly filtered fields (e.g., date, region)
- Determine if users repeatedly query overlapping data ranges without partition pruning
- Assess whether queries experience slow performance or high latency for recent data
How to fix
6 ways to remove the waste.
- Convert raw data to columnar formats such as Parquet or ORC to reduce scan size
- Partition data based on common query dimensions (e.g., date, tenant ID)
- Consolidate small files into larger batches to improve scan efficiency
- Adopt table formats like Apache Iceberg, Delta Lake, or Hudi for metadata support and schema evolution
- Stream data into these formats using structured pipelines to reduce latency and support efficient querying
- Refactor query patterns or create materialized views to reduce redundant scans
Documentation
Vendor references for pricing and configuration.