Ingesting thousands of IoT sensors without losing data
Sustainergy runs on data from thousands of energy sensors reporting continuously. The hard part was never the volume - it was the gaps, the duplicates, and the sensors that lie. Lessons from the ingestion layer.
Sustainergy (see the full IoT & analytics case study) is a facility and asset intelligence platform. Strip away the dashboards and the recommendations and the ML, and underneath it all is one job that has to be done right: take readings from thousands of energy sensors, continuously, and never quietly lose any.
It sounds like a volume problem. It is not, really. The volume is steady and predictable. The problem is that physical sensors in the real world are unreliable narrators.
What the ingestion layer has to survive
Sensors drop offline and come back. They report late, in bursts, after a network blip. They send the same reading twice. They send a reading with a timestamp from 1970, or from next week. They occasionally send a value that is physically impossible. And all of this is normal — it is not an error condition, it is Tuesday.
So the ingestion layer is not "receive data and store it". It is "receive a messy, unreliable, out-of-order stream and turn it into something the rest of the platform can trust".
How we built it
Sensors publish over MQTT, which is the right call for this — lightweight, built for flaky connections, designed for exactly this kind of fan-in. From the broker, readings flow into a pipeline that does a few things in order:
- Validate. Is this reading physically possible? Is the timestamp sane? A reading that fails validation is not dropped silently — it is quarantined, so we can see that a sensor is misbehaving and how.
- De-duplicate. The same reading arriving twice should not become two data points. Every reading has a natural key — sensor plus timestamp — and writes are idempotent on that key.
- Order. Readings arrive out of order constantly. The storage layer is built to accept late-arriving data and slot it into the correct place in the timeline, rather than assuming data arrives in the order it was measured.
Only after all that does a reading count as "ingested".
Where it bit us
Late data broke the aggregates. Sustainergy shows customers consumption rolled up by hour, by day, by month. Our first version computed those rollups as data came in. Then a sensor that had been offline for six hours reconnected, dumped six hours of backdated readings, and every rollup that touched that window was now wrong — and nothing knew to recompute it. We had to make aggregation aware that the past is not finished. Late data triggers a recompute of the affected windows. The rollup is a view of the data, not a one-time event.
"Gap" and "zero" are not the same thing, and the difference matters. A sensor reporting zero consumption and a sensor reporting nothing are completely different facts, and early on we treated them the same. That quietly corrupted everything downstream — a customer's "savings" looked great because a sensor had simply gone dark. We made absence an explicit, first-class state. A gap is a gap. It is visible, it is flagged, and no downstream calculation gets to pretend it was a zero.
The broker was a single point of failure we did not respect. Everything fanned in through MQTT, and we treated the broker as infrastructure that just works — until it did not, and we had no buffer between "broker hiccup" and "data loss". We added a durable buffer so that a momentary broker or pipeline problem means data is delayed, not gone.
What we would tell you
If you are building IoT ingestion, the mental model that helped us most was this: you are not receiving data, you are reconstructing a timeline from unreliable witnesses.
- Validate at the edge, and quarantine instead of dropping — a bad reading is a signal about a sensor.
- Make idempotency non-negotiable. Duplicate delivery is the default, not the exception.
- Treat late data as normal and make your aggregates recomputable.
- Make "no data" a real state. It is not a zero. It is not noise. It is information.
The volume will not be the thing that hurts you. The sensor that has been confidently reporting nonsense for two weeks — that is the thing that hurts you.
Working on something similar?
We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.