Historical data gets revised. Your backtest doesn't know that.
You backtest a strategy, get a result, and move on. Months later you pull the same history and the numbers have changed - because history changed. On Weatherstone, designing for a past that is not fixed was its own problem.
There is a comfortable assumption baked into most backtesting code: the past is finished. You ran the strategy over 2019 to 2023, you got a number, and that number is the truth about that period forever.
It is not. On Weatherstone we kept running into the same uncomfortable fact — the historical data we backtested against last month is not necessarily the historical data we get when we pull the same range this month. The past gets edited.
Why history moves
Market and fundamental data gets revised after the fact, for ordinary reasons. A data provider corrects an error. A corporate action gets applied retroactively. A fundamental figure is restated. A source backfills a gap it used to have. None of this is unusual or wrong — it is just how data sources work. But the consequence is real: a backtest is a function of its input data, and if the input data changes, the backtest result changes, even though not a line of strategy code moved.
The first time this bites you, it is genuinely disorienting. A strategy you validated is now showing different numbers and nothing you control has changed. Either you can explain it or you cannot trust any of your results, including the ones that still look fine.
What we built
Snapshotting. When a backtest runs, we capture the exact data it ran against — not a reference to "the 2019-2023 data", but the actual values, as they were on the day of the run. A result is stored together with its input snapshot. That makes any past result reproducible: you can re-run it and get the same number, because you still have the same data.
Revision detection. The data pipeline does not just pull the latest data and overwrite. It compares what it just pulled against what it had before, and when historical values have changed, that is an event — logged, visible, attributable to a date range and a source. We stopped finding out about revisions by accident and started being told about them.
Re-validation on revision. When a revision lands in a range that a stored strategy depends on, that strategy gets flagged for a re-run. The old result is not silently replaced and it is not silently kept — both are kept, side by side, with the note that the underlying data changed between them. A strategy whose performance shifts meaningfully when the data is corrected is telling you something important about how fragile it is.
Separating "data changed" from "strategy changed". Because every result carries its data snapshot and its strategy version, we can always answer the only question that matters when a number moves: was that us, or was that the data? Without that, every surprising result turns into a guessing game.
The pitfall
The trap is not the revision itself. The trap is a backtesting setup that always pulls "the latest data" and never records what that data was. It feels fine — it is even arguably more correct, you are always using the best available data — right up until you cannot reproduce a result you reported, cannot explain why a number changed, and have no way to tell a data revision apart from a bug you introduced. You have built a system whose outputs you cannot stand behind.
What we would tell you
If you are backtesting anything, treat the input data as a versioned artifact, not as a live feed you re-read each time.
- Snapshot the data with the result. A result you cannot reproduce is not a result.
- Detect revisions explicitly. History changing should be an event, not a surprise.
- Re-validate what a revision touches, and keep both the before and the after.
- Always be able to answer "data or strategy". If you cannot, every anomaly costs you a day.
The past is not fixed. The least you can do is record which version of it you were looking at when you made a decision.
Working on something similar?
We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.