# Inefficient file format and layout for Athena queries Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/inefficient-file-format-and-layout-for-athena-queries Storing raw JSON or CSV files in S3 - especially when written frequently in small batches-leads to excessive scan costs in Athena. By: Erik Marke Updated: 2026-09-28 [Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) # Inefficient File Format and Layout for Athena Queries The short version Storing raw JSON or CSV files in S3 - especially when written frequently in small batches-leads to excessive scan costs in Athena. [Erik Marke](https://www.linkedin.com/in/erikmarke/) Product Owner IT Financial Management & FinOps @ Swiss Post AWS service [AWS Athena](https://www.pointfive.co/efficiency-hub/cloud-services/aws-athena) Category [Compute](https://www.pointfive.co/efficiency-hub/service-category/compute) Reference CER-0188 Type Inefficient Architecture ## Explanation Why the waste happens and who it affects. These formats are row-based and verbose, requiring Athena to scan and parse the full content even when only a few fields are queried. Without columnar formats, partitioning, or metadata-aware table formats, queries become inefficient and expensive, especially in high-volume environments. ## Billing model The pricing dimensions that drive this cost. Pay-per-scan - Athena charges based on the amount of data scanned per query, not the size of the result set. This makes storage format, partitioning, and file layout critical cost drivers. ## How to detect 5 checks to find it in your estate. - Review whether data stored in S3 is in raw formats like JSON or CSV - Check if files are written at high frequency, resulting in large numbers of small files - Evaluate whether data is partitioned by commonly filtered fields (e.g., date, region) - Determine if users repeatedly query overlapping data ranges without partition pruning - Assess whether queries experience slow performance or high latency for recent data ## How to fix 6 ways to remove the waste. - Convert raw data to columnar formats such as Parquet or ORC to reduce scan size - Partition data based on common query dimensions (e.g., date, tenant ID) - Consolidate small files into larger batches to improve scan efficiency - Adopt table formats like Apache Iceberg, Delta Lake, or Hudi for metadata support and schema evolution - Stream data into these formats using structured pipelines to reduce latency and support efficient querying - Refactor query patterns or create materialized views to reduce redundant scans ## Documentation Vendor references for pricing and configuration. - [Optimize Athena performance docs.aws.amazon.com](https://docs.aws.amazon.com/athena/latest/ug/performance-tuning.html) - [Use columnar storage formats docs.aws.amazon.com](https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html) - [Amazon Athena Pricing aws.amazon.com](https://aws.amazon.com/athena/pricing/) ## Related inefficiencies [Browse the library](https://www.pointfive.co/efficiency-hub) - AWS Athena CER-0136 ### [Unmanaged Growth of Athena Query Output Buckets](https://www.pointfive.co/efficiency-hub/inefficiencies/unmanaged-growth-of-athena-query-output-buckets) When a workgroup writes results to a customer-managed S3 location, Athena generates a new S3 object for every query result, regardless of whether the output is needed long term. Over time, this leads to uncontrolled growth of the output... Storage - AWS Lambda CER-0207 ### [Recursive Invocation Loop Between Lambda and SQS](https://www.pointfive.co/efficiency-hub/inefficiencies/recursive-invocation-loop-between-lambda-and-sqs) When a Lambda function processes messages from an SQS queue but fails to handle certain messages properly, the same messages may be returned to the queue and retried repeatedly. In some cases, especially if the Lambda is also writing... Compute - AWS EC2 CER-0088 ### [Suboptimal Region for Internet-Only EC2 Instance](https://www.pointfive.co/efficiency-hub/inefficiencies/suboptimal-region-for-internet-only-ec2-instance) When an EC2 instance is dedicated primarily to internet-facing traffic, regional differences in data transfer pricing can drive a substantial portion of total costs. Hosting such workloads in a region with higher egress rates leads to... Compute --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.