AI & Agents·4 min read·April 18, 2024

Trending is easy. On-brand and usable is the hard part.

For Pockla we had to surface trending moments for brands. The catch: most trends are either irrelevant to the brand or impossible to make content from. Here is how we taught the system to tell the difference.

When we started working with Pockla (see the full case study), the brief sounded simple enough: find trending content and turn it into posts brands can publish. A week in, we realised "trending" was the least interesting word in that sentence.

The actual problem

Pockla is a content platform. Marketers come to it expecting two things: tell me what is hot right now, and help me make something out of it. The naive version of this is a trending feed — scrape social platforms, rank by velocity, show the top results. We built that in a few days and it was, honestly, useless.

Here is why. A trend has to clear two completely separate bars before it is worth a marketer's time:

  1. Relevance — does this have anything to do with the brand? A fitness brand does not care that a celebrity divorce is trending.
  2. Usability — even if it is relevant, can you actually make content from it? Some moments are just news. There is no angle, no hook, nothing for the brand to do.

A feed sorted purely by velocity gets both of these wrong constantly. It surfaces loud, generic noise and buries the quiet, specific moment that the brand could have owned.

What we built

We stopped thinking of it as a feed and started thinking of it as a filter with three stages.

Stage one: a content pool. We scrape and normalise content from a lot of sources — social platforms, news, niche communities — into one store. This pool is deliberately broad. We are not filtering yet, just collecting and de-duplicating, because you cannot filter for relevance against something you have not seen.

Stage two: a brand profile. Before we score anything, an agent builds a profile of the brand — what it sells, who it talks to, the tone it uses, the topics it has touched before, the topics it would never touch. This profile is the thing relevance gets measured against. We wrote about that profiling agent separately because it turned into its own small project.

Stage three: scoring for both bars. Every candidate moment from the pool gets scored twice. Once for relevance to the brand profile, using embeddings plus an LLM pass for the judgement calls embeddings are bad at. And once for what we ended up calling "creatability" — can a marketer actually build a post from this? The second score is the one everyone forgets, and it is the one that made the product feel smart.

Where it went wrong

The first version scored relevance well and creatability badly. The model was happy to tell us a moment was "highly usable" and then, when you asked it for an actual content angle, produce something nobody would ever post.

The fix was not a better prompt. The fix was to make the model show its work. Instead of asking "is this usable, score 1-10", we asked it to draft the actual angle — the hook, the format, the first line — and then we scored the moment on whether that draft was any good. If the model could not produce a usable angle, the moment was not usable, by definition. Creatability stopped being an opinion and became an artifact we could check.

The second problem was drift. Brand profiles built once and never updated slowly went stale — a brand pivots, runs a campaign, changes its voice, and the profile is still scoring against who they were six months ago. We made the profile something that gets refreshed on a schedule and also nudged whenever the brand's own published content starts diverging from it.

What we would tell you

If you are building anything that "finds relevant content", the lesson from Pockla is that relevance is usually two or three questions wearing a trench coat. Pull them apart. Score each one separately. And whenever you can, make the model produce the artifact you actually care about — the draft, the answer, the recommendation — and judge that, instead of asking the model to rate a thing in the abstract. Models are far better at doing the work than at grading it.

The trending feed, by the way, is still in there. It is just three filters down from where it started.

AI AgentsRAGContent GenerationPocklaLLM

Working on something similar?

We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.