When one backtest's output is another's input
Single strategies are tidy. The moment one strategy consumes another's output, you have a pipeline - and on Weatherstone, managing that web of dependencies became the difference between trustworthy results and a tangle.
A single strategy is a manageable thing. It has inputs, it has logic, it produces a result, you reason about it end to end. The work on Weatherstone stopped being tidy the moment strategies started depending on each other — one strategy's output became another strategy's input, and then a third's, and what we had was no longer a set of strategies but a pipeline.
That shift is easy to miss while it is happening, and expensive to ignore once it has happened.
How it sneaks up on you
It starts innocently. You build a strategy that produces a signal, or a baseline, or a constructed series. It is good. Someone builds a second strategy that, reasonably, uses the first one's output instead of recomputing it. Then a third uses the second. Each step is sensible on its own. But now you have a directed graph, and a graph has properties that a list does not.
If strategy A's output changes — because its logic changed, or because, as covered in the post on data revisions, its input data was revised — then B is now stale, and C, which depends on B, is stale too, and nothing about B or C announces this. They will happily run again and produce confident numbers from inputs that no longer reflect A.
What we did about it
Modelled the graph explicitly. Every strategy declares what it consumes. A strategy that depends on another strategy's output declares that, the same way it declares a dependency on a data source. The system knows the dependency graph rather than it living in people's heads.
Versioned every output. A strategy's output is not just "the latest result". It is a versioned artifact tagged with the strategy version and the input snapshot that produced it. When B consumes A's output, it consumes a specific version of it, and it records which one. That is what makes the whole thing reproducible — you can always reconstruct exactly which version of A fed which run of B.
Made staleness propagate. When A is re-run and its output changes, everything downstream of A is automatically marked stale. Not re-run necessarily — marked stale, visibly, so nobody is looking at a C result built on an A that has since moved. Re-running the affected subtree in dependency order is then a single deliberate action.
Ordered execution by the graph. Running the pipeline is not "run all the strategies". It is a topological walk of the graph — A before B before C — so nothing ever runs before its inputs are ready. Obvious in hindsight. Not obvious when it grew one strategy at a time.
The pitfalls
Circular dependencies, introduced by accident. Someone wires strategy C's output back into something A consumes, usually without realising the full chain, and now you have a cycle that either loops or deadlocks. Making the graph explicit meant we could simply reject a cycle at definition time instead of discovering it at run time.
Diamond dependencies and inconsistent inputs. B and C both depend on A, and D depends on both B and C. If B and C ran against different versions of A, then D is combining two things that disagree about reality. Versioned outputs fixed this — D can require that its inputs trace back to a consistent version of shared ancestors.
The debugging cost of an implicit graph. Before we modelled it, a wrong number in C meant manually tracing backward through B and A and their data, by hand, every time. That is hours. With an explicit, versioned graph, "what did this depend on, and what changed" is a query, not an investigation.
What we would tell you
The moment one piece of analysis consumes another's output, you have a pipeline, and pipelines need to be treated like pipelines:
- Make the dependency graph explicit. If it only exists in people's heads, it is already wrong somewhere.
- Version outputs, not just code. Downstream consumers should depend on a specific version, not "the latest".
- Propagate staleness. A changed input should visibly invalidate everything downstream of it.
- Execute in dependency order. Topological, every time. Never "just run everything".
You can build the graph deliberately now, or you can reverse-engineer it later during an incident. We have done both. The first one is cheaper.
Working on something similar?
We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.