Data Engineering·4 min read·August 15, 2024

Thousands of calculations a second, and what happens when one fails

Sustainergy turns raw sensor data into insights through a web of calculations that depend on each other. Designing for the failure of any single one - without taking down the rest - was the real work.

Once sensor data is safely ingested, Sustainergy (see case study) has to do something with it. That something is a large set of calculations — consumption rollups, anomaly checks, savings estimates, recommendations — running continuously, thousands of them a second. And they are not independent. One calculation's output is another's input. A savings estimate depends on a baseline, which depends on a clean consumption series, which depends on a dozen sensor readings being valid.

So the question that defined this part of the project was never "how do we make it fast". It was "what happens when one calculation in the middle of that web fails".

A dependency graph, whether you admit it or not

The first thing we had to be honest about: this is a dependency graph. We could pretend it was a list of jobs, but the moment calculation B reads calculation A's result, A and B have an edge between them, and the system has a shape whether we modelled it or not.

So we modelled it. Each calculation declares what it depends on. The system knows the graph. That sounds like overhead, and it is, but it buys you the thing that makes everything else possible: when something fails, you know exactly what is downstream of it.

The failure modes we had to design for

A dependency is not ready yet. Calculation B runs, but A has not produced its result. The wrong answer here is for B to run anyway on stale or missing input — that is how a small gap becomes a confidently wrong number on a customer's dashboard. The right answer is for B to wait, or to be scheduled only once its inputs exist. We made dependency state explicit: a calculation is not eligible to run until everything it depends on is fresh.

A dependency failed outright. A bad sensor poisons a consumption series, the rollup that needs it fails. Everything downstream of that rollup should not fail too, and it should definitely not run on garbage. We made failure propagate as a state, not as a crash. A calculation whose input is in a failed state goes into a "blocked" state itself — it has not errored, it simply cannot run yet, and it will run the moment its input recovers.

A calculation succeeds slowly. The sneaky one. A calculation that takes ten times longer than usual has not failed, so nothing alerts, but everything waiting on it is now late, and the delay compounds down the graph. We had to monitor latency per calculation, not just success and failure. "It worked" and "it worked in time" are different questions.

Recovery has to be cheap. When a sensor's data gets corrected — and it does, constantly — every calculation downstream of it needs to re-run. If recovery means manually kicking off jobs, recovery does not happen. So re-running a subtree of the graph is a first-class operation: point at the calculation that changed, and everything downstream of it re-runs in dependency order, automatically.

What we got wrong first

Our first version retried failed calculations in place, immediately, with no awareness of why they failed. A calculation blocked on missing input would retry, fail because the input was still missing, retry again, and burn resources doing it. Retrying is the right instinct for a transient error and exactly the wrong one for a dependency that is not ready. Separating "this failed, try again" from "this cannot run yet, wait" was the single change that made the system calm down.

What we would tell you

If you have calculations that depend on each other, you have a dependency graph. You can either model it deliberately or discover it during an incident.

  • Make dependencies explicit. The system should know its own shape.
  • Failure is a state, not an exception. Let it propagate as "blocked", so downstream work waits instead of running on bad data.
  • Monitor latency, not just success. Slow is a failure mode too.
  • Make re-running a subtree trivial. Data gets corrected. If recovery is manual, it will not happen.

The goal is not a system where nothing ever fails. It is a system where one failure stays one failure, instead of becoming a thousand.

Data PipelinesDistributed SystemsSustainergyReliability

Working on something similar?

We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.