Terabytes on AWS Glue: every failed run costs you twice
Migrating multiple terabytes through AWS Glue sounds like a configuration task. It is not. When a run takes hours and a failure near the end means starting over, the economics change how you have to build.
We have run a lot of large data migrations through AWS Glue. The work for Candu.ai (see the full data migration case study) — moving terabytes of historical analytics data into a database built for fast queries — is the one that taught us the most, mostly by going wrong in instructive ways.
The naive view of a Glue job is that it is a script. You write the PySpark, you point it at the source, you run it. The reality, at terabyte scale, is that you are running a process that takes hours, costs real money per hour, and where a failure ninety percent of the way through can mean you start again from zero. That changes everything about how you should build it.
Why the economics matter
A small job that fails is annoying. You fix it, you re-run it, you have lost minutes. A terabyte-scale Glue job that fails is expensive twice over: you paid for the hours of compute that led up to the failure, and you will pay for them again on the re-run. Do that three or four times while you are still finding bugs and the migration's cost is dominated by failed attempts, not successful ones.
So the goal is not "write a job that works". The goal is "write a job where a failure is cheap".
What that looks like in practice
Make the job restartable, not re-runnable. A re-runnable job starts from the beginning every time. A restartable job knows what it has already done and picks up from there. We broke the migration into partitions — by time range, mostly — tracked which partitions had completed successfully, and made re-running the job skip the ones that were already done. A failure now costs you one partition, not the whole dataset.
Make it idempotent. Restartability only works if re-processing a partition is safe. Writing the same partition twice has to produce the same result as writing it once, or your retries create duplicates and you have traded a crash for silent corruption, which is a worse trade.
Test on data that is actually representative — and assume it is not. This is the one that cost us the most, and it gets its own post, but the short version: we tested on a sample, the sample looked clean, the full dataset was not, and we found out hours into a run. Sample data is a useful smoke test and a dangerous source of confidence.
Right-size deliberately, then watch it. Glue's worker count and type is a real cost-versus-speed dial, and the correct setting is not guessable from the job's code — it depends on data shape, skew and what the job actually does. We stopped guessing, ran measured trials on partitions, and watched Spark for the usual villains: skew dumping all the work on one worker, a join exploding, a shuffle spilling to disk. A job that is twice as slow as it should be is a job you are paying twice as much for.
The pitfalls, condensed
- Failure near the end is the expensive case. Design so that the cost of a failure is one partition, not the whole run.
- "It worked on the sample" is not "it works". The full dataset has cases your sample did not.
- Retries without idempotency create corruption. That is strictly worse than the crash you were trying to avoid.
- Cost is a function of job design, not just data size. A badly partitioned job at terabyte scale is a budget problem, not a performance footnote.
What we would tell you
Before you run a large Glue migration, ask one question: what does a failure three hours in cost me, and how do I make that number small? If the honest answer is "the whole run", stop and restructure before you start. Partition the work, make each partition idempotent and skippable, and track completion. The first run will still find bugs — it always does. The point is to make finding them cheap.
The migration for Candu shipped, the historical data landed where it needed to be, and the queries that used to be slow got fast. But if we are honest, the lasting value was the discipline: at this scale, you are not writing a script, you are designing for the failure case. The success case mostly takes care of itself.
Working on something similar?
We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.