What it changed
- Sensor data processed in real time, as it arrives, instead of in batches.
- Ingestion that scales with the number of devices, without a server to resize.
- Spikes and failures absorbed by queues instead of lost.
The situation
The company's devices sit in paddocks across Australia, measuring the conditions farmers make decisions on. They report all day and all night, and they don't report politely: connections drop, devices reconnect and send hours of backlog at once, and a whole fleet can wake up in the same minute. The data was useful only if it arrived quickly and completely.
What I did
I helped the company scale its ingestion and process the data in real time. The design principles are the ones I'd apply to any device fleet: accept every reading the moment it arrives, put a queue between the devices and everything else so a spike can never knock over the database, process in batches so the cost stays sane, and make every step safe to retry so a failure never loses or duplicates a reading.
The result is a pipeline that scales with the number of devices, absorbs the bad days, and turns sensor data into something the business and its customers can act on straight away. The general pattern, with the numbers, is in the IoT ingestion guide.
Why it matters to you
Real-time is easy in a demo with ten devices. It gets hard at thousands of devices, unreliable networks and a bill that must stay sensible. That's where experience with queues, back pressure and idempotency is the difference between a system that works and one that loses data quietly.