# Amazon Data Firehose

> The cheapest way to land a stream of records in S3. No servers, no consumers to write, and about $0.04/GB in Sydney.

- Verdict: My default
- Vendor: AWS
- Layer: [APIs, queues and events](https://cloudlesslabs.com/stack/layers/api-events.md). Job: Managed delivery of records into S3 (and other sinks)
- Last reviewed: 19 October 2026, by Nicolas Dao (Cloudless Labs, Sydney)
- Web page: https://cloudlesslabs.com/stack/aws-firehose/. The whole stack in one file: https://cloudlesslabs.com/stack/brief.md

Firehose is one of AWS's quiet workhorses. You hand it records, it batches, compresses, converts and writes them to S3, and retries for you. Logs are the obvious use, but anything that should end up as files in the data lake fits.

## Use it when

- Chatty application logs: the function's logger batches DEBUG/INFO lines into Firehose, which writes gzip or Parquet files to S3. About $75 a month per TB vs about $673 on CloudWatch.
- Streaming events, clicks, device readings or audit records into the data lake, without writing a consumer.
- Anything that should end up as files in S3, batched, compressed and retried for you.

## Don't use it when

- Work that needs processing per message, retries per message or a dead-letter queue: that's SQS with Lambda.
- Several consumers replaying the same stream: that's Kinesis Data Streams.
- Logs you need to watch live or alarm on: keep WARN and ERROR in CloudWatch.

## Pricing

Sydney, Direct PUT: $0.037/GB ingested (first 500 TB), billed per record in 5 KB steps. JSON to Parquet conversion $0.023/GB. Dynamic partitioning $0.025/GB plus per-object fees (avoidable). Then normal S3 storage.

## Gotchas, with fixes

- **Gotcha:** Direct PUT bills every record rounded up to 5 KB. One 200-byte log line per record costs 25 times the list price, about $925 per TB.
  **Fix:** Pack many newline-delimited lines into each record (up to about 1,000 KB). Records of 100 KB or more make the rounding negligible.
- **Gotcha:** Lambda freezes the environment when the handler returns, so buffered lines can be lost.
  **Fix:** Await a flush before returning (try/finally), flush on size, and send WARN/ERROR synchronously to console so CloudWatch keeps the critical lines.
- **Gotcha:** A stream accepts 1 MiB/s by default in Sydney. Busy peaks throttle.
  **Fix:** Request a higher limit up front, and retry the records reported in FailedPutCount.
- **Gotcha:** Lambda's built-in log destination to Firehose is not the cheap path: CloudWatch charges $0.335/GB to deliver it, then Firehose charges $0.17/GB for vended logs.
  **Fix:** Call PutRecordBatch from your own logger instead.
- **Gotcha:** Dynamic partitioning looks handy and adds about $25 per TB plus per-object and per-hour fees.
  **Fix:** Use the default YYYY/MM/DD/HH prefix (free) and one stream per app or per table.

## Provisioning

Official Pulumi package · Official Terraform provider

@pulumi/aws kinesis.FirehoseDeliveryStream with destination extended_s3.

## Instead, consider

- [Amazon CloudWatch](https://cloudlesslabs.com/stack/aws-cloudwatch/index.md): Side by side, not instead: CloudWatch for the critical lines you watch live; Firehose for the chatty bulk.
- [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md): Each message needs work and retries.
- [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md): Several consumers must replay the same ordered stream.

## Worked examples

- [CloudWatch and Firehose, side by side](https://cloudlesslabs.com/stack/guides/chatty-logs/index.md): How do I keep chatty logs without a CloudWatch-sized bill?
- [Athena or DuckDB?](https://cloudlesslabs.com/stack/guides/data-lake/index.md): How should I query my data lake, and what changed when AWS bought DuckDB's company?

## Where it fits

- [Event-driven plumbing](https://cloudlesslabs.com/stack/use-cases/events.md): pick 4. Streams into S3: records batched, compressed and written as files for the data lake, with no consumer to write.
- [A data lake](https://cloudlesslabs.com/stack/use-cases/data-lake.md): pick 2. The way in. Streams records from apps and devices into S3, batched and compressed.
- [Logs without a shocking bill](https://cloudlesslabs.com/stack/use-cases/logging.md): pick 2. The chatty bulk, side by side: your logger batches DEBUG/INFO into Firehose, which writes to S3. About $75 per TB instead of about $673.

## Sources

- [Amazon Data Firehose pricing](https://aws.amazon.com/firehose/pricing/)
- [Firehose quotas](https://docs.aws.amazon.com/firehose/latest/dev/limits.html)
- [Converting input record format (Parquet)](https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html)

## Related

- [APIs, queues and events](https://cloudlesslabs.com/stack/layers/api-events.md): every tool in this layer, with its verdict
- [The whole stack in one file](https://cloudlesslabs.com/stack/brief.md): every use case and tool, for questions that span many
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)