Your sample data is lying to you
On a terabyte-scale migration for Candu, the sample data looked clean and the full dataset did not. The bug you find in production is the cheap one - the data-quality bug you find three hours into a run is the expensive one.
Here is a sequence of events we have lived through more than once, told in its purest form by the Candu migration.
You are moving terabytes of historical data from one store to another. You cannot test against terabytes every time you change a line, so you pull a sample. The sample looks clean. Your transformation handles it perfectly. You feel good. You kick off the full run. Three hours later it fails — or worse, it succeeds, and the data is subtly wrong — because the full dataset contained shapes your sample did not.
Now you fix the transformation, and re-run from the start, and pay for those three hours again. And then it fails on a different shape. This is the loop that eats data migration budgets.
Why the sample lies
A sample is, by construction, the common case. Random sampling gives you a representative slice of the typical data, and the typical data is exactly the part that was never going to break your pipeline. The records that break pipelines are the rare ones — the unusual encodings, the nulls in fields that are "never" null, the legacy rows written by a system that had different rules, the duplicates, the timestamps from before the company existed.
By definition, a small random sample mostly misses these. The cleaner your sample looks, the more confident you are, and the more confident you are, the harder the full run hits.
What we do now
Sample for the weird, not just the typical. Alongside the random sample, we deliberately go looking for the edges: the oldest records, the newest, the biggest, the ones with nulls, the ones from known-legacy sources, anything that looks like an outlier. A sample chosen to find problems is worth ten times a sample chosen to be representative.
Profile the full dataset before transforming any of it. Before writing the real transformation, we run a cheap pass over the entire dataset that does not transform anything — it just counts. How many nulls in each field, what is the range of each value, how many distinct formats appear in this column, what is the cardinality. This is fast and cheap even at terabyte scale because it is just aggregation, and it tells you what you are actually dealing with instead of what your sample suggested.
Make the pipeline report what it could not handle. The transformation does not get to silently skip a record it does not understand, and it does not get to crash on it either. Anything it cannot confidently handle goes to a quarantine output with a reason attached. After a run, the quarantine is a to-do list: here are the shapes we did not anticipate, here is how many of each. You fix the pipeline against real, named cases instead of guessing.
Validate the output, not just the run status. A Glue job that finishes is not a Glue job that did the right thing. We reconcile after every run — row counts, key checks, aggregate totals that should match between source and destination. On the Candu migration this is exactly what caught the issues that did not crash anything: the run was green, the numbers did not tie out, and that mismatch was the bug.
The honest part
We did not do all of this from the start. We did it because the first version of the migration cost us re-runs, and re-runs at that scale are slow and they are not free. The data-quality issues were not exotic — they were the normal sediment that any dataset with a few years of history accumulates. The mistake was trusting a clean-looking sample and finding out the hard way, on the clock.
What we would tell you
For any migration or large transformation:
- Profile the whole dataset first. It is cheap. Do it before you write the transformation, not after it fails.
- Sample to find problems, not to feel reassured. Hunt the edges on purpose.
- Quarantine, do not skip and do not crash. Turn "unexpected data" into a list you can work through.
- Reconcile the output. "The job finished" and "the data is correct" are different claims. Check the second one.
The data-quality bug is going to exist either way. Your only real choice is whether you find it cheaply, in a profiling pass, or expensively, three hours into a run you are paying for.
Working on something similar?
We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.