# Real-time ingestion for a national farm-sensor network

> Thousands of field devices reporting around the clock, processed as the data arrives instead of hours later, on infrastructure that scales with the network.

- Client: An Australian agriculture-technology company with a large IoT sensor network (Agriculture technology)
- Role: Architect and lead engineer, Nicolas Dao
- Status: Delivered. At Cloudless Labs
- Capabilities: IoT, Real-time data, Event-driven architecture, Serverless
- Web page: https://cloudlesslabs.com/work/realtime-iot/

## Outcomes

- Sensor data processed in real time, as it arrives, instead of in batches.
- Ingestion that scales with the number of devices, without a server to resize.
- Spikes and failures absorbed by queues instead of lost.

## The situation

The company's devices sit in paddocks across Australia, measuring the conditions farmers make decisions on. They report all day and all night, and they don't report politely: connections drop, devices reconnect and send hours of backlog at once, and a whole fleet can wake up in the same minute. The data was useful only if it arrived quickly and completely.

## What I did

I helped the company scale its ingestion and process the data in real time. The design principles are the ones I'd apply to any device fleet: accept every reading the moment it arrives, put a queue between the devices and everything else so a spike can never knock over the database, process in batches so the cost stays sane, and make every step safe to retry so a failure never loses or duplicates a reading.

The result is a pipeline that scales with the number of devices, absorbs the bad days, and turns sensor data into something the business and its customers can act on straight away. The general pattern, with the numbers, is in the [IoT ingestion guide](https://cloudlesslabs.com/stack/guides/iot-ingestion/index.md).

## Why it matters to you

Real-time is easy in a demo with ten devices. It gets hard at thousands of devices, unreliable networks and a bill that must stay sensible. That's where experience with queues, back pressure and idempotency is the difference between a system that works and one that loses data quietly.

## Built with

- [Amazon Web Services (AWS)](https://cloudlesslabs.com/stack/aws/index.md): My default in The Stack
- [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md): My default in The Stack
- [AWS Lambda](https://cloudlesslabs.com/stack/aws-lambda/index.md): My default in The Stack
- Real-time ingestion
- Back pressure

## Related

- [IoT ingestion into Postgres](https://cloudlesslabs.com/stack/guides/iot-ingestion/index.md): the decision guide behind it
- [Cutting an AWS bill from US$240k a year to under US$40k](https://cloudlesslabs.com/work/aws-bill-cut/index.md): a related project
- [From legacy ASP.NET and stored procedures to Node.js on serverless](https://cloudlesslabs.com/work/dotnet-to-serverless/index.md): a related project
- [Crypto market analysis and an automated trading bot](https://cloudlesslabs.com/work/crypto-analysis/index.md): a related project
- [An e-commerce platform built from scratch in California](https://cloudlesslabs.com/work/ecommerce-from-scratch/index.md): a related project
- [Every project](https://cloudlesslabs.com/work/index.md)
- [How we work](https://cloudlesslabs.com/about/how-we-work/index.md)
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)