# CloudWatch and Firehose, side by side

> How do I keep chatty logs without a CloudWatch-sized bill? Not either-or. WARN and ERROR stay in CloudWatch, where you watch and alarm on them. The chatty bulk goes from your logger to Firehose to S3, at about $75 per TB instead of about $673.

- A decision guide from The Stack by Nicolas Dao (Cloudless Labs, Sydney)
- Web page: https://cloudlesslabs.com/stack/guides/chatty-logs/. The whole stack in one file: https://cloudlesslabs.com/stack/brief.md
- Tools: [Amazon CloudWatch](https://cloudlesslabs.com/stack/aws-cloudwatch/index.md) (My default), [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md) (My default), [Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md) (My default), [DuckDB](https://cloudlesslabs.com/stack/duckdb/index.md) (Use when it fits), [S3 + Glue + Athena](https://cloudlesslabs.com/stack/aws-athena-data-lake/index.md) (Use when it fits)

**The two jobs are different.** During an incident you want live logs, alarms and metric filters: that's CloudWatch. For the chatty DEBUG and INFO lines you want them kept, cheaply, and searchable when someone asks "what happened to order 1234 last Tuesday?": that's S3.

```
Lambda ─┬─ console.warn / console.error ──▶ CloudWatch   (live, alarms, ~2% of volume)
        └─ log.debug / log.info ─ batched ─▶ Firehose ─▶ S3 (gzip or Parquet) ─▶ Athena / DuckDB
```

## What 1 TB of chatty logs costs a month (Sydney)

| Path | Monthly |
|---|---|
| CloudWatch Standard ($0.67/GB) | ~$673 |
| CloudWatch Infrequent Access ($0.335/GB) | ~$342 |
| Lambda's built-in S3 destination ($0.335/GB delivery) | ~$338 |
| Lambda's built-in Firehose destination ($0.335 + $0.17/GB) | ~$508 |
| **Your logger → Firehose → S3, batched** | **~$75** (Parquet conversion adds ~$29) |
| Your logger → Firehose, one line per record | ~$925 (the 5 KB trap) |

Add about $13 for the WARN/ERROR lines that stay in CloudWatch. The ~$75 assumes about 100M invocations with one ~10 KB record each, including about $27 of extra Lambda time for the flush. With bigger records it drops to $45–70.

**Why the built-in destination isn't the cheap path:** CloudWatch still handles the delivery and charges $0.335/GB in Sydney for it. Your own logger skips CloudWatch entirely and pays Firehose's $0.037/GB.

## Routing: per function, or per line

- **Per function, no code:** each function's logging config points at a log group. Critical functions stay on Standard, a verbose one can go to an Infrequent Access group. This doesn't split lines inside one function.
- **Per line, a small logger:** WARN and ERROR to `console`, everything else to a buffer flushed to Firehose. This is the split that saves the money.

## The logger (sketch, about 40 lines)

```js
import { FirehoseClient, PutRecordBatchCommand } from '@aws-sdk/client-firehose'

const firehose = new FirehoseClient({})
const STREAM = process.env.LOG_STREAM
const MAX_RECORD = 900 * 1024 // Firehose bills each record in 5 KB steps: pack lines together
let lines = []

const line = (level, msg, data) => JSON.stringify({ ts: new Date().toISOString(), level, msg, ...data })

export const log = {
	debug: (msg, data) => lines.push(line('DEBUG', msg, data)),
	info: (msg, data) => lines.push(line('INFO', msg, data)),
	warn: (msg, data) => console.warn(line('WARN', msg, data)),   // CloudWatch
	error: (msg, data) => console.error(line('ERROR', msg, data)), // CloudWatch
}

export async function flush() {
	if (!lines.length) return
	const records = []
	let cur = ''
	for (const l of lines) {
		if (cur && Buffer.byteLength(cur) + Buffer.byteLength(l) > MAX_RECORD) { records.push(cur); cur = '' }
		cur += l + '\n'
	}
	if (cur) records.push(cur)
	lines = []
	for (let i = 0; i < records.length; i += 4) { // max 4 MiB per call
		const res = await firehose.send(new PutRecordBatchCommand({
			DeliveryStreamName: STREAM,
			Records: records.slice(i, i + 4).map(r => ({ Data: Buffer.from(r) })),
		}))
		if (res.FailedPutCount) console.warn(line('WARN', 'log records dropped', { failed: res.FailedPutCount }))
	}
}

// In each handler:
export const handler = async event => {
	try {
		log.info('received', { id: event.id })
		// ...
	} finally {
		await flush() // Lambda freezes after return: flush first
	}
}
```

For production, add a size-based flush, a flush shortly before the timeout (`context.getRemainingTimeInMillis()`), and a retry of failed records. Don't use an async pino transport here: Lambda freezes it mid-send.

## The infrastructure (Pulumi)

```js
const logs = new aws.kinesis.FirehoseDeliveryStream('api-logs', {
	destination: 'extended_s3',
	extendedS3Configuration: {
		roleArn: firehoseRole.arn,        // may write to the bucket
		bucketArn: lake.arn,
		prefix: 'logs/app=api/',          // Firehose appends YYYY/MM/DD/HH/ for free
		errorOutputPrefix: 'logs-errors/',
		bufferingSize: 64,                // MB
		bufferingInterval: 300,           // seconds
		compressionFormat: 'GZIP',
	},
})
// The function's role needs firehose:PutRecordBatch on logs.arn,
// and the bucket a lifecycle rule that expires logs/ after your retention.
```

## Searching them

From a laptop or a Lambda, with DuckDB:

```sql
SELECT ts, level, msg
FROM read_json('s3://lake/logs/app=api/2026/10/19/*/*.gz')
WHERE id = 'order-1234'
ORDER BY ts;
```

Or define a Glue table over `logs/` with partition projection on the date and query it from Athena.

## The trade-offs, honestly

- **No Live Tail, no alarms** on the S3 side. That's why WARN and ERROR stay in CloudWatch.
- **A small loss window:** lines still in memory are lost on a crash or timeout. The error itself still reaches CloudWatch.
- **A few to tens of milliseconds** per invocation for the flush.
- **Throughput:** 1 MiB/s per stream by default in Sydney. Ask for more before a busy launch.
- **Cut volume first anyway:** JSON log format, no payloads, retention on every log group. Cheap logs are still logs nobody reads.

## Related

- [Logs without a shocking bill](https://cloudlesslabs.com/stack/use-cases/logging.md): the use case this guide works through
- [Amazon CloudWatch](https://cloudlesslabs.com/stack/aws-cloudwatch/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md): the full entry (verdict, pricing, gotchas)
- [DuckDB](https://cloudlesslabs.com/stack/duckdb/index.md): the full entry (verdict, pricing, gotchas)
- [S3 + Glue + Athena](https://cloudlesslabs.com/stack/aws-athena-data-lake/index.md): the full entry (verdict, pricing, gotchas)
- [Every decision guide](https://cloudlesslabs.com/stack/brief.md): listed at the end of the brief
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)