# A data lake

> Analytics over everything the apps produce, cheaply, without a warehouse to run. Nicolas Dao's picks for this job (Cloudless Labs, Sydney), in order of preference, what he would avoid, and what to use instead.

- Use case 13 of 21 in The Stack, last reviewed 21 October 2026
- Web page: https://cloudlesslabs.com/stack/use-cases/data-lake/

Picks, in order of preference:

1. **[Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md)** (My default): The lake. Parquet (Iceberg next), partitioned by date and sorted by your second key.
2. **[Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md)** (My default): The way in. Streams records from apps and devices into S3, batched and compressed.
3. **[S3 + Glue + Athena](https://cloudlesslabs.com/stack/aws-athena-data-lake/index.md)** (Use when it fits): Big scans, ad-hoc questions and reports. Zero code, $5 per TB scanned.
4. **[DuckDB](https://cloudlesslabs.com/stack/duckdb/index.md)** (Use when it fits): App queries, writing Parquet, local dev and tests, on the same files. AWS bought the company behind it in August 2026.
5. **[Aurora PostgreSQL](https://cloudlesslabs.com/stack/aws-aurora/index.md)** (Use when it fits): If the app already runs on Aurora: join live tables with the lake in one SQL query (DuckDB inside).

Worked example: [Athena or DuckDB?](https://cloudlesslabs.com/stack/guides/data-lake/index.md)

## The picks in brief

- [Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md): My default. No complaints. Storage, data lake, archive. Pricing: Sydney: $0.025/GB-month (Standard). Internet egress $0.114/GB (first 10 TB): that's where it hurts. Requests: $5.50 per million PUT/LIST, $0.44 per million GET. Standard-IA $0.0138/GB, Glacier Instant Retrieval $0.005/GB. Transfer from S3 to CloudFront and to AWS compute in the same region is free.
- [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md): My default. The cheapest way to land a stream of records in S3. No servers, no consumers to write, and about $0.04/GB in Sydney. Pricing: Sydney, Direct PUT: $0.037/GB ingested (first 500 TB), billed per record in 5 KB steps. JSON to Parquet conversion $0.023/GB. Dynamic partitioning $0.025/GB plus per-object fees (avoidable). Then normal S3 storage.
- [S3 + Glue + Athena](https://cloudlesslabs.com/stack/aws-athena-data-lake/index.md): Use when it fits. The data lake that works. Not amazing. Pair it with DuckDB. Pricing: Sydney: Athena $5 per TB scanned, 10 MB minimum per query. S3 Standard $0.025/GB-month. Glue Data Catalog free for the first million objects.
- [DuckDB](https://cloudlesslabs.com/stack/duckdb/index.md): Use when it fits. SQL over Parquet that runs anywhere, from my laptop to a Lambda. Next to Athena, not instead of it. Pricing: Free and MIT-licensed. In Lambda you pay compute and S3 GETs: a query that reads 1 GB costs about $0.002, against $0.005 on Athena, with no 10 MB minimum per query.
- [Aurora PostgreSQL](https://cloudlesslabs.com/stack/aws-aurora/index.md): Use when it fits. Peace of mind and scale, at a price. My choice when the client must stay all-AWS. Pricing: Serverless v2 kept at 0.5 ACU always-on ≈ $73 a month. RDS Proxy adds at least about $26 a month (provisioned) or about $146 (Serverless v2, minimum 8 ACU).

## Related

- [Athena or DuckDB?](https://cloudlesslabs.com/stack/guides/data-lake/index.md): the worked example for this job, with costs
- [APIs, queues and events](https://cloudlesslabs.com/stack/layers/api-events/index.md): every tool in this layer
- [Data](https://cloudlesslabs.com/stack/layers/data/index.md): every tool in this layer
- [Storage and delivery](https://cloudlesslabs.com/stack/layers/storage/index.md): every tool in this layer
- [Every use case and tool in one file](https://cloudlesslabs.com/stack/brief.md)
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)