AI & Agents·4 min read·June 13, 2024

Building a content pool you can actually generate from

Pockla's content engine runs on a pool of scraped, processed content. Keeping that pool fresh, relevant and small enough to query fast turned into the real engineering problem behind the AI.

The AI parts of Pockla (see the full case study) get the attention — the agents, the generation, the scoring. But the thing that actually decides whether the product works is much less glamorous: a pool of content that everything else queries against. Get the pool wrong and the smartest agent in the world is generating from garbage.

What the pool is for

Pockla generates content for brands. To do that well it needs a large, fresh body of source material — trending moments, reference posts, examples of what is working right now in a brand's space. That body of material is the pool. The brand-profiling agent reads from it, the relevance scorer queries it, the generation step pulls examples from it.

So the pool has three jobs at once: be broad enough to contain the relevant thing, be fresh enough that "trending" still means trending, and be fast enough to query that none of the agents are sitting around waiting.

Those three jobs fight each other.

The freshness problem

Trending content has a shelf life measured in days, sometimes hours. A pool that is broad but stale is just an archive. Our first version treated the pool like a database — write once, keep forever — and within a couple of weeks the most "relevant" results it returned were weeks old.

We moved to treating the pool as a window, not an archive. Content has a recency weight, old content ages out of the active pool, and the scraping pipeline is biased toward keeping the window full rather than keeping everything. Permanence is a nice property for a system of record. It is the wrong property for a trending engine.

The relevance-vs-noise problem

A broad pool is a noisy pool. Vector search over a noisy pool returns things that are near your query but not useful — semantically similar, practically irrelevant. We were getting a lot of "technically related" results that no marketer would ever act on.

Two things helped. First, we did more filtering at write time, not just read time — content that is obviously low quality or off-topic never makes it into the pool in the first place, so search is not fighting it later. Second, we stopped trusting vector similarity as the final answer. Similarity gets you a candidate set; an LLM pass over that candidate set makes the actual judgement call. Embeddings are a great coarse filter and a bad final judge.

The pitfalls nobody warns you about

De-duplication is harder than it looks. The same moment shows up across a dozen sources, slightly reworded each time. Exact-match de-duplication catches none of it. We de-duplicate on semantic similarity with a tuned threshold, and tuning that threshold is a genuine trade-off — too loose and you collapse distinct moments together, too strict and the pool fills up with fifteen copies of the same thing crowding out everything else.

Vector store growth is silent until it is not. A vector store that you only ever insert into gets slower and more expensive in a way that does not announce itself. It just gradually degrades. Because we had moved to a windowed pool, ageing content out solved this almost by accident — but if we had kept the archive model, we would have hit a wall. Decide your retention policy early. It is much harder to retrofit.

The embedding model is a dependency, not a detail. Change the embedding model and every vector in the store is now in a different space than every new vector you write. We learned to treat the embedding model as a versioned, pinned dependency, and to plan re-embedding as a real migration rather than a config change.

What we would tell you

If you are building a RAG system, the retrieval layer is the product. The model on top is interchangeable; the pool is not. Spend your time on:

  • A clear retention policy. Decide what ages out and when, before the store gets big.
  • Filtering at write time. It is cheaper to keep junk out than to out-rank it later.
  • Similarity as a filter, not a verdict. Use it to shortlist. Use a model to decide.

The pool is the least exciting part of Pockla to talk about and the part we would rebuild first if we were doing it again. That is usually how it goes.

RAGAI AgentsPocklaChromaDBVector Search

Working on something similar?

We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.