Data Engineering·4 min read·July 25, 2024

Scraping the internet for a content engine, without it quietly falling over

Behind Pockla's AI is a scraping pipeline that has to keep a content pool full from sources that change, rate-limit and break constantly. Celery, RabbitMQ, and a long list of lessons about backpressure.

Pockla's content pool only works if something keeps it full. That something is a scraping and processing pipeline that pulls from a long list of sources, normalises what it finds, and feeds it into the pool. It runs on Python, Django, Celery and RabbitMQ, and for the first month it broke in a different way roughly every week.

None of the breakages were dramatic. That was the problem.

What the pipeline does

At a high level it is boring: fetch content from a source, clean and normalise it, run it through processing, write it to the pool. Multiply that by many sources, each with its own format, rate limits and reliability, and run it continuously so the pool never goes stale.

We split it into stages and put each stage on its own Celery queue — fetching, processing, writing — so a slow source could not block a fast one and we could scale each stage independently. That part worked well from day one. The trouble was everywhere else.

The failures, in order of how much they cost us

Silent partial failure. A source changes its layout. The scraper does not crash — it just starts returning empty results, or half a result. The task succeeds. The queue drains. Everything looks healthy on the dashboard. Meanwhile that source has contributed nothing to the pool for three days. We had built a pipeline that could fail without telling anyone.

The fix was to make "success" mean something. A task no longer succeeds just because it did not throw. It has to return a plausible result — a non-trivial amount of content, in the expected shape — or it fails loudly. We also added per-source health: if a source's output drops off a cliff, that is an alert, not a shrug.

Retries that made things worse. Celery makes retries easy, which means it makes retry storms easy. A source goes down, every task hammering it fails, every failed task retries, the retries fail, and now we are issuing far more requests to a struggling source than we ever would have normally — and getting ourselves rate-limited or blocked for the trouble. We added exponential backoff with jitter, a cap on retry attempts, and a circuit breaker per source so that a source that is clearly down gets left alone for a while instead of getting pounded.

No backpressure. When processing fell behind, the fetch stage cheerfully kept going, piling work into RabbitMQ faster than it could be drained. The queue depth grew, memory grew, and the failure when it came was sudden and ugly. We added backpressure: the fetch stage checks downstream queue depth and slows itself down when processing is behind. A pipeline that cannot push back on its own input is a pipeline waiting to fall over.

Tasks that were not idempotent. A task that gets retried — or delivered twice, which RabbitMQ will absolutely do — has to be safe to run again. Some of ours were not, so retries created duplicate content in the pool, which then made the de-duplication step work harder than it should have. We made every task idempotent: writing the same scraped item twice produces the same result as writing it once.

What we would tell you

A scraping pipeline is not really a scraping problem. It is a distributed-systems problem wearing a scraping costume. The scraping part is easy. The hard part is everything around it:

  • Define success. A task that did not crash is not a task that worked. Make success mean "produced a real result".
  • Retries need limits and backoff. An unbounded retry is a denial-of-service attack you are running against yourself.
  • Backpressure is not optional. If upstream cannot tell that downstream is behind, you will find out when something runs out of memory.
  • Assume every task runs at least twice. Build for it. It will.

The pipeline still breaks sometimes — sources change, the internet is the internet. The difference now is that it breaks loudly, in one place, and tells us which source and why. That is the whole goal. You are not aiming for a pipeline that never fails. You are aiming for one that fails in a way you can act on.

Data PipelinesCeleryRabbitMQPocklaWeb Scraping

Working on something similar?

We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.