Idempotency, retries and backfills: pipelines you can trust
Every data pipeline we have built eventually taught us the same three lessons. They are not advanced. They are just the difference between a pipeline you can rerun without fear and one you cannot.
We have built data pipelines for a content engine, for thousands of IoT sensors, for terabyte-scale migrations, for quant backtesting. Different domains, different scales, different tools. They all eventually taught us the same three things, and they are not advanced things. They are just the things that separate a pipeline you can rerun without fear from one that makes you nervous every time you touch it.
One: idempotency is not optional
An idempotent pipeline stage produces the same result whether it runs once or five times. This sounds like a nice-to-have. It is actually the property that everything else depends on.
Here is why it is not optional: your pipeline will run things more than once. A task will be retried. A message queue will deliver the same message twice — not might, will. A backfill will overlap with a range that already ran. An operator will rerun a job because they were not sure if it finished. If re-processing the same input creates duplicate output, then every one of those normal, expected events is now a data-corruption event.
The fix is almost always the same: every record has a natural key, and writes are upserts on that key rather than blind inserts. Processing the same record twice overwrites itself instead of duplicating itself. Once a pipeline is idempotent, rerunning it stops being scary, and a pipeline you can rerun without fear is a pipeline you can actually operate.
Two: retries need backoff, limits and judgement
Retrying is the right response to a transient failure. It is the wrong response to a few other things, and a pipeline that retries indiscriminately turns a small problem into a large one.
A retry with no backoff, when a downstream service is struggling, is just you helping it fall over faster. A retry with no limit, against a failure that is not transient, is an infinite loop burning money. And a retry of a "failure" that is actually "my input is not ready yet" is wasted work that should have been a wait instead.
So retries need three things: backoff with jitter, so a struggling dependency gets room to recover instead of a stampede; a cap, so a permanent failure becomes a visible dead-letter rather than an infinite loop; and a distinction between "failed" and "not ready", so that the second one waits politely instead of retrying pointlessly. A retry policy is a real design decision. Treating it as a default checkbox is how retry storms happen.
Three: design for backfills before you need one
You will need to reprocess history. The logic changed and yesterday's output is now wrong. A bug got fixed and a week of data needs redoing. A source corrected its data and everything downstream is stale. Backfilling is not an edge case — it is a routine, recurring operation, and a pipeline that cannot do it cleanly makes every one of those situations a crisis.
A backfillable pipeline has a few properties. It is parameterised by time range — you can point it at any window, not just "now". Its stages are idempotent, so a backfill safely overwrites instead of duplicating (which is lesson one, again — they compound). And it can run a backfill without fighting the live pipeline, so reprocessing March does not break today.
If backfilling is designed in, it is a routine command. If it is not, it is a tense afternoon of manual work every single time, and you will have many of those afternoons.
How they compound
These are not three separate tips. They reinforce each other. Idempotency is what makes retries safe — a retried task that is idempotent cannot corrupt anything. Idempotency is also what makes backfills safe, for the same reason. And sane retry behaviour is what keeps a backfill of a large range from turning into a self-inflicted denial-of-service. Get idempotency right and the other two get much easier. Skip it and the other two are quietly dangerous.
What we would tell you
Before a data pipeline goes anywhere near production, ask three questions:
- If this stage runs twice, is the result the same as running it once? If not, fix that first. It is the foundation.
- What happens when something downstream is down — does this back off, give up sensibly, and tell someone? Or does it stampede?
- Can I reprocess an arbitrary past window with one command, without breaking the present? If not, you do not have a pipeline, you have a pipeline plus a recurring bad afternoon.
None of this is clever. All of it is the difference between infrastructure you trust and infrastructure you tiptoe around.
Working on something similar?
We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.