# S3 + Glue + Athena

> The data lake that works. Not amazing. Pair it with DuckDB.

- Verdict: Use when it fits
- Vendor: AWS
- Layer: [Data](https://cloudlesslabs.com/stack/layers/data.md). Job: Data lake
- Last reviewed: 18 October 2026, by Nicolas Dao (Cloudless Labs, Sydney)
- Web page: https://cloudlesslabs.com/stack/aws-athena-data-lake/. The whole stack in one file: https://cloudlesslabs.com/stack/brief.md

Parquet in S3, catalogued by Glue, queried by Athena. Reliable and cheap at rest. Athena stays for the big scans; DuckDB takes the app queries, the Parquet writing and local development. Iceberg is next on my list to dig into.

## Use it when

- Analytics on large volumes of Parquet in S3, including data exported from Durable Objects and apps.
- Ad-hoc questions, scheduled reports and scans of tens of GB to TBs: Athena fans out, queues and scales with zero code.

## Don't use it when

- An app endpoint firing many small queries: each costs at least 10 MB, takes seconds and needs polling. Use DuckDB in a Lambda.

## Pricing

Sydney: Athena $5 per TB scanned, 10 MB minimum per query. S3 Standard $0.025/GB-month. Glue Data Catalog free for the first million objects.

## In my words: what hurts

- Works, but it's not amazing.
- I have to design a clear schema and save everything as Parquet.
- One index is easy (the partition key). A second one means duplicating the data.

## Gotchas, with fixes

- **Gotcha:** Data partitioned by date, queried by customer: every query scans every partition.
  **Fix:** Keep the date partitions and sort each file by the second key (DuckDB COPY ... ORDER BY). Athena and DuckDB skip row groups by their min/max statistics, and Athena bills only the bytes it reads. For a real fix, use Iceberg tables in S3 Tables, partitioned by day(ts) plus bucket(N, customer_id), with sort or z-order compaction.
- **Gotcha:** Partition projection and Glue partition indexes look like secondary indexes. They only speed up query planning.
  **Fix:** Use sorting, bucketing or Iceberg to prune data; projection to skip catalog lookups.
- **Gotcha:** CTAS and INSERT INTO write at most 100 partitions per query.
  **Fix:** Write Parquet with DuckDB COPY (no limit, full control of sort order and file size), or split the job.

## Provisioning

Official Pulumi package · Official Terraform provider

## Instead, consider

- [DuckDB](https://cloudlesslabs.com/stack/duckdb/index.md): App queries, writing Parquet, and local dev. Side by side with Athena.

## Worked examples

- [CloudWatch and Firehose, side by side](https://cloudlesslabs.com/stack/guides/chatty-logs/index.md): How do I keep chatty logs without a CloudWatch-sized bill?
- [Athena or DuckDB?](https://cloudlesslabs.com/stack/guides/data-lake/index.md): How should I query my data lake, and what changed when AWS bought DuckDB's company?

## Where it fits

- [A data lake](https://cloudlesslabs.com/stack/use-cases/data-lake.md): pick 3. Big scans, ad-hoc questions and reports. Zero code, $5 per TB scanned.

## In The Stack Weekly

- [Week 43: Aurora PostgreSQL reads Iceberg and Parquet](https://cloudlesslabs.com/weekly/2026-w43/index.md), worth a pilot

## Sources

- [Athena pricing](https://aws.amazon.com/athena/pricing/)
- [Athena performance tuning (data layout)](https://docs.aws.amazon.com/athena/latest/ug/performance-tuning-data-optimization-techniques.html)
- [S3 Tables maintenance (compaction)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html)

## Related

- [Data](https://cloudlesslabs.com/stack/layers/data.md): every tool in this layer, with its verdict
- [The whole stack in one file](https://cloudlesslabs.com/stack/brief.md): every use case and tool, for questions that span many
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)