A fire alarm at 2:14 am

5,000 fire and gas detectors across 200 sites in Australia, all reporting telemetry through the IoT ingestion design. At 2:14 am one of them measures smoke. The promise to the client: someone on the phone within a minute.

I haven't built a life-safety platform. This is the design I'd build, worked through from the IoT pipeline I have built, and checked against the services' documentation and prices.

• The cloud's share is a few seconds. The rest belongs to the network and the people, so design those parts first.
• Split alarms from telemetry at the front door, and never let a fire wait behind readings.
• Count only a keypress as "someone knows". A sent SMS, a delivered push or a call that reached voicemail doesn't.
• Prove it every week with a drill that goes all the way to a phone.

The design

 detector ── decides: smoke ── local sounder
    │ MQTT, one connection, two topics
    ├── $aws/rules/readings/...  ──▶ readings rule ──▶ SQS (batched) ──▶ Lambda ──▶ Postgres
    │                                                                    (the IoT ingestion guide)
    └── $aws/rules/alarms/...    ──▶ alarms rule ── error action: second queue
                                       ▼
                                 SQS alarm queue ──▶ Lambda (batch 1, reserved concurrency)
                                                        ├──▶ PagerDuty (the client's on-call team)
                                                        └──▶ Twilio (anyone who isn't a seat)
                                                        then ──▶ Postgres: the record, after the call

The detector decides. Detecting smoke is its whole job, so the cloud doesn't re-derive it from readings. It publishes alarms to their own Basic Ingest topic, and IoT Core routes them to their own queue without a broker or a Lambda in between. Cloud-side rules still have a job, for alerts no single device can see: a gas level rising for an hour, three detectors at one site, a device gone silent. Those run on the telemetry path, which is fine: they're minutes-scale by nature.

Who answers the phone

That question picks the service. Paging services charge per person who can be paged; a voice API charges per call.

So: PagerDuty for the client's own team, Twilio when the people called are site managers or customers. Many clients need both: the site manager first through Twilio, the operations team through PagerDuty if nobody acknowledges.

AWS's own voice service (End User Messaging) doesn't fit: its calls are one-way text-to-speech, with no "press 1", and voice is capped at 20 calls a minute.

What it costs

The alarm path is nearly free: a few dozen alarms a month through IoT Core, SQS and Lambda round to cents, and reserved concurrency costs nothing. The real lines are the people-facing ones: PagerDuty seats (about $21–25 per person a month), Twilio calls ($0.075 a minute to Australian mobiles, plus about $8.25 a month per number), and a handful of CloudWatch alarms. Next to the stakes, it's the cheapest part of the platform; spend the effort on the network and the drills instead.

The gotchas

• Account-wide concurrency. Every Lambda function shares one pool, 1,000 by default. A reconnect storm on the telemetry side can throttle the alarm function. Give it reserved concurrency (20 is plenty), and raise the account quota before a big fleet goes live.
• One fire, fifty calls. A spreading fire trips several detectors, and QoS 1 resends. Set PagerDuty's dedup_key to the site and the hazard, or keep one open incident per site in your own code.
• Do Not Disturb. At night, SMS, email and push are silenced or unread. Push only breaks through with Apple's Critical Alerts permission, which Apple grants rarely. Call first; text as the record.
• Voicemail answers. A call that reaches voicemail reports as answered. Only a keypress acknowledges. Turn on answering-machine detection to move to the next person sooner.
• Branded SMS in Australia. Since 1 July 2026, an unregistered sender name shows as "Unverified". Send from an Australian mobile number instead: it's unaffected, and people can reply.
• Sleeping radios. Battery devices on LTE-M or NB-IoT sleep between readings. The firmware must wake the radio the moment it detects smoke, not at the next reporting slot. Test it on the real network.
• False offline alarms. Lifecycle events can arrive twice or out of order. Raise "detector offline" from missed heartbeats after a grace period, never from a single disconnect event, and raise one site-level incident when a whole site goes quiet.
• The database in the way. Writing the incident to Postgres before calling fails exactly when Postgres is slow or down. Call first, record after.
• A quiet night or a broken pipeline. They look the same. Alarm on the alarm queue's ApproximateAgeOfOldestMessage and on the function's errors and throttles, with missing data treated as breaching.

The sketch

The alarm queue, the function and the wiring, in Pulumi:

const alarmDlq = new aws.sqs.Queue('alarms-dlq', { messageRetentionSeconds: 14 * 86400 })
const alarmQueue = new aws.sqs.Queue('alarms', {
	visibilityTimeoutSeconds: 60, // ≥ 6 × the 10 s function timeout
	redrivePolicy: pulumi.jsonStringify({ deadLetterTargetArn: alarmDlq.arn, maxReceiveCount: 10 }),
})

new aws.iot.TopicRule('alarms', {
	name: 'alarms', // devices publish to $aws/rules/alarms/<thing name>
	enabled: true,
	sql: 'SELECT *, clientid() AS deviceId, timestamp() AS receivedAt',
	sqlVersion: '2016-03-23',
	sqs: [{ queueUrl: alarmQueue.url, roleArn: iotRole.arn, useBase64: false }],
	errorAction: { sqs: { queueUrl: alarmDlq.url, roleArn: iotRole.arn, useBase64: false } },
})

const alarm = new aws.lambda.Function('alarm', {
	/* runtime, code, role… */
	timeout: 10,
	reservedConcurrentExecutions: 20, // no backlog elsewhere can take this capacity
})

new aws.lambda.EventSourceMapping('alarms', {
	eventSourceArn: alarmQueue.arn,
	functionName: alarm.name,
	batchSize: 1, // and no batching window
})

The function, paging the client's team through PagerDuty's Events API:

import type { SQSEvent } from 'aws-lambda'

type Alarm = { siteId: string; siteName: string; hazard: string; deviceId: string; ts: string }

export const handler = async (event: SQSEvent) => {
	const a: Alarm = JSON.parse(event.Records[0].body)
	const res = await fetch('https://events.pagerduty.com/v2/enqueue', {
		method: 'POST',
		headers: { 'content-type': 'application/json' },
		body: JSON.stringify({
			routing_key: process.env.PD_ROUTING_KEY, // from Secrets Manager
			event_action: 'trigger',
			dedup_key: `${a.siteId}:${a.hazard}`, // one incident per site and hazard
			payload: { summary: `${a.hazard} detected at ${a.siteName}`, source: a.deviceId, severity: 'critical', timestamp: a.ts, custom_details: a },
		}),
	})
	if (!res.ok) throw new Error(`PagerDuty ${res.status}`) // the message returns to the queue and is retried
	await record(a).catch(err => console.error('record failed', err)) // after the call, never before
}

Without PagerDuty, the same function calls through Twilio and sets the escalation timer: one EventBridge Scheduler schedule per step, deleted by the keypress webhook.

await twilio.calls.create({
	to: contact.phone,
	from: process.env.ALERT_NUMBER, // an Australian mobile number
	twiml: `<Response><Gather numDigits="1" action="${ACK_URL}?incident=${id}">
		<Say>${xmlEscape(`${a.hazard} detected at ${a.siteName}.`)} Press 1 to acknowledge.</Say>
	</Gather></Response>`,
	machineDetection: 'Enable',
})
await scheduler.send(new CreateScheduleCommand({
	Name: `escalate-${id}-${step}`,
	ScheduleExpression: `at(${inTwoMinutes})`, // yyyy-mm-ddThh:mm:ss
	FlexibleTimeWindow: { Mode: 'OFF' },
	ActionAfterCompletion: 'DELETE',
	Target: { Arn: ESCALATE_FN_ARN, RoleArn: SCHEDULER_ROLE_ARN, Input: JSON.stringify({ id, step: step + 1 }) },
}))

The keypress webhook validates Twilio's signature, records who acknowledged, and deletes the pending schedule. The escalation function checks the incident is still unacknowledged before calling the next person. It's a short loop in code, which is why Step Functions isn't needed here.

Drills

Every week, a scheduled job publishes a drill alarm from a test device through the real path, to a test rota or the real one in business hours. The acknowledgement writes a metric, and a CloudWatch alarm fires when it's missing. Once a quarter, unplug a site's gateway and time how long "site offline" takes to reach a phone.

When this design is wrong

If the client needs a certified, monitored fire alarm, this isn't it: that system belongs to the fire-protection contractor and its monitoring service, and this design alerts people alongside it. That's also where my work stops: I'll build the cloud and alerting side (the alarm path, the calls, the escalation, the drills and the monitoring of all of it), working alongside the client's certified system, and never design or certify the alarm system itself. If the sites lose internet often, the cloud can't help: the alarm needs a second path on site, such as a cellular dialler on the gateway, before any of this matters. And if a missed hour of AWS Sydney is unacceptable, that second path is also the cheapest answer, cheaper and simpler than a second region.