Firehose is one of AWS's quiet workhorses. You hand it records, it batches, compresses, converts and writes them to S3, and retries for you. Logs are the obvious use, but anything that should end up as files in the data lake fits.
Use it when
- Chatty application logs: the function's logger batches DEBUG/INFO lines into Firehose, which writes gzip or Parquet files to S3. About $75 a month per TB vs about $673 on CloudWatch.
- Streaming events, clicks, device readings or audit records into the data lake, without writing a consumer.
- Anything that should end up as files in S3, batched, compressed and retried for you.
Don't use it when
- Work that needs processing per message, retries per message or a dead-letter queue: that's SQS with Lambda.
- Several consumers replaying the same stream: that's Kinesis Data Streams.
- Logs you need to watch live or alarm on: keep WARN and ERROR in CloudWatch.
Pricing
Sydney, Direct PUT: $0.037/GB ingested (first 500 TB), billed per record in 5 KB steps. JSON to Parquet conversion $0.023/GB. Dynamic partitioning $0.025/GB plus per-object fees (avoidable). Then normal S3 storage.
Gotchas, with fixes
Gotcha: Direct PUT bills every record rounded up to 5 KB. One 200-byte log line per record costs 25 times the list price, about $925 per TB.
Fix: Pack many newline-delimited lines into each record (up to about 1,000 KB). Records of 100 KB or more make the rounding negligible.
Gotcha: Lambda freezes the environment when the handler returns, so buffered lines can be lost.
Fix: Await a flush before returning (try/finally), flush on size, and send WARN/ERROR synchronously to console so CloudWatch keeps the critical lines.
Gotcha: A stream accepts 1 MiB/s by default in Sydney. Busy peaks throttle.
Fix: Request a higher limit up front, and retry the records reported in FailedPutCount.
Gotcha: Lambda's built-in log destination to Firehose is not the cheap path: CloudWatch charges $0.335/GB to deliver it, then Firehose charges $0.17/GB for vended logs.
Fix: Call PutRecordBatch from your own logger instead.
Gotcha: Dynamic partitioning looks handy and adds about $25 per TB plus per-object and per-hour fees.
Fix: Use the default YYYY/MM/DD/HH prefix (free) and one stream per app or per table.
Provisioning
Official Pulumi package · Official Terraform provider
@pulumi/aws kinesis.FirehoseDeliveryStream with destination extended_s3.
Instead, consider
- Amazon CloudWatch Side by side, not instead: CloudWatch for the critical lines you watch live; Firehose for the chatty bulk.
- Amazon SQS Each message needs work and retries.
- Amazon Kinesis Several consumers must replay the same ordered stream.
Worked examples
Where it fits
- Event-driven plumbing: Pick 04.
Streams into S3: records batched, compressed and written as files for the data lake, with no consumer to write.
- A data lake: Pick 02.
The way in. Streams records from apps and devices into S3, batched and compressed.
- Logs without a shocking bill: Pick 02.
The chatty bulk, side by side: your logger batches DEBUG/INFO into Firehose, which writes to S3. About $75 per TB instead of about $673.
Sources
Last reviewed by Nicolas Dao