When your database bill grows faster than your data
An IoT platform generates data forever and deletes none of it. On Sustainergy we watched storage costs climb until we had to get serious about what data is worth keeping hot, and what is not.
Sustainergy ingests data from thousands of sensors and never stops. Every day the dataset is bigger than the day before, and none of it ever gets less true, so the instinct is to keep all of it, in the database, ready to query. That instinct is expensive.
About a year in, the storage line on the cloud bill was growing faster than the customer base. That is not a sustainable shape for a business, so we had to actually think about data the way an accountant thinks about inventory: not all of it is worth the same, and not all of it should be stored the same way.
The realisation
We were treating every reading identically. A sensor reading from this morning and a sensor reading from fourteen months ago lived in the same place, in the same format, on the same expensive, query-optimised storage. But those two readings are not used the same way at all.
This morning's reading gets queried constantly — live dashboards, current anomaly checks, this month's rollups. The fourteen-month-old reading gets queried almost never, and when it does, it is almost always as part of an aggregate, not on its own.
We were paying hot prices for cold data, at scale, forever.
What we changed
Tiering by temperature. Recent data stays hot — fast storage, fully indexed, instantly queryable. Older data moves to a colder, cheaper tier. It is still there, still correct, still queryable; it is just not pretending to be data anyone needs in single-digit milliseconds. The move happens automatically based on age.
Cheaper formats for cold data. Raw rows are a fine format for data you are constantly writing and updating. They are a wasteful format for historical data that will never change again. Cold data gets stored in a compact, compressed columnar format, which is both far cheaper to store and, conveniently, far better for the kind of large aggregate scans that historical data actually gets used for.
Rollups as a retention strategy. Here is the part that took us longest to accept: for old data, we do not always need every individual reading. If a customer is looking at energy consumption from eighteen months ago, an hourly rollup answers the question. So beyond a certain age, we keep the rollups at full fidelity and thin out the raw readings underneath them. The questions people actually ask of old data are answered just as well, for a fraction of the cost.
The pitfalls
Tiering is easy to get into and hard to reason about later. Once data lives in three places, "query the data" becomes "query the data, wherever it happens to be". We had to make that boundary invisible to the rest of the platform — a query layer that knows where data lives and fetches it from the right tier — otherwise every feature would have to know about the storage strategy, and that is a mess that spreads.
Down-sampling is a one-way door. Once you have thinned out raw readings, they are gone. Before we turned that on, we had to be genuinely sure that nothing — no current feature, no plausible near-future feature, no compliance need — required raw historical data at full resolution. Tiering is reversible. Deletion is not. We were conservative about the line between them, and we should have been.
Cost problems hide until they do not. The thing about storage cost is that it never spikes. It creeps. There is no incident, no alert, no bad day — just a number that is slightly bigger every month until one day someone looks at it properly. We added the boring thing we should have had from the start: cost as a monitored metric, with the trend visible, so the conversation happens early instead of in a panic.
What we would tell you
If you are building anything that accumulates data indefinitely — IoT, logs, events, analytics — decide your data lifecycle before the bill makes you.
- Tier by how data is actually used, not by how old it is for its own sake. Age is just a good proxy for usage.
- Match the format to the access pattern. Columnar and compressed for cold aggregate scans. Rows for hot reads and writes.
- Treat down-sampling as deletion, because it is. Be sure.
- Monitor cost as a metric. It will never alert you on its own.
Storage feels free when you are small. It is one of those costs that is genuinely fine to ignore right up until it is the most important number on the bill.
Working on something similar?
We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.