Parquet in S3, catalogued by Glue, queried by Athena. Reliable and cheap at rest. Athena stays for the big scans; DuckDB takes the app queries, the Parquet writing and local development. Iceberg is next on my list to dig into.
Use it when
- Analytics on large volumes of Parquet in S3, including data exported from Durable Objects and apps.
- Ad-hoc questions, scheduled reports and scans of tens of GB to TBs: Athena fans out, queues and scales with zero code.
Don't use it when
- An app endpoint firing many small queries: each costs at least 10 MB, takes seconds and needs polling. Use DuckDB in a Lambda.
Pricing
Sydney: Athena $5 per TB scanned, 10 MB minimum per query. S3 Standard $0.025/GB-month. Glue Data Catalog free for the first million objects.
In my words: what hurts
- Works, but it's not amazing.
- I have to design a clear schema and save everything as Parquet.
- One index is easy (the partition key). A second one means duplicating the data.
Gotchas, with fixes
Gotcha: Data partitioned by date, queried by customer: every query scans every partition.
Fix: Keep the date partitions and sort each file by the second key (DuckDB COPY ... ORDER BY). Athena and DuckDB skip row groups by their min/max statistics, and Athena bills only the bytes it reads. For a real fix, use Iceberg tables in S3 Tables, partitioned by day(ts) plus bucket(N, customer_id), with sort or z-order compaction.
Gotcha: Partition projection and Glue partition indexes look like secondary indexes. They only speed up query planning.
Fix: Use sorting, bucketing or Iceberg to prune data; projection to skip catalog lookups.
Gotcha: CTAS and INSERT INTO write at most 100 partitions per query.
Fix: Write Parquet with DuckDB COPY (no limit, full control of sort order and file size), or split the job.
Provisioning
Official Pulumi package · Official Terraform provider
Instead, consider
- DuckDB App queries, writing Parquet, and local dev. Side by side with Athena.
Worked examples
Where it fits
- A data lake: Pick 03.
Big scans, ad-hoc questions and reports. Zero code, $5 per TB scanned.
Sources
Last reviewed by Nicolas Dao