Quant & Fintech·4 min read·November 14, 2024

Backtesting on free market data: garbage in, confident garbage out

For Weatherstone's quant work, the strategies were only ever as trustworthy as the data underneath them. Free market data is full of quiet problems, and a backtest will happily turn those problems into a number that looks like a result.

Weatherstone (see the full quantitative systems case study) does quantitative analysis and portfolio work — collecting market data, building strategies, backtesting them, reporting on what works. Early on, the temptation with any project like this is to spend all your energy on the strategy logic, because that is the interesting part. The strategy logic was not where the risk was. The data was.

A backtest is a machine for turning historical data into a confident-looking number. If the data has problems, the backtest does not warn you. It just gives you a worse number, or a better one, with exactly the same confidence. That is the trap.

What is wrong with free data

Free and low-cost market data sources are genuinely useful and we use them. But they come with a standard set of problems, and you have to know them or they will find you:

  • Gaps. Missing days, missing instruments, missing fields. A naive backtest treats a gap as "nothing happened", which is rarely what actually happened.
  • Survivorship bias. Many free datasets only contain instruments that still exist. The ones that went to zero — exactly the ones a risk model needs to see — have quietly been removed from history. Backtest on that and every strategy looks safer than it is.
  • Silent revisions. Historical values get corrected after the fact. Pull the same date range twice, months apart, and you can get different numbers. We have a whole separate post on this one because it deserves it.
  • Adjustments done inconsistently. Splits, dividends and other corporate actions are sometimes applied, sometimes not, sometimes applied twice. A price series that mixes adjusted and unadjusted data produces returns that never happened.

None of these throw an error. All of them change your results.

What we built around it

A validation layer between the data and the backtest. Raw data from any source does not go straight into a strategy. It goes through checks first: continuity (are there gaps, and are they real market closures or missing data), sanity (are these values physically plausible), and consistency (does this series agree with itself and, where we can, with a second source). Data that fails gets flagged, not silently used.

Provenance on every series. We track where each piece of data came from and when we pulled it. When a backtest produces a surprising result, the first question is always "is this the strategy or is this the data", and you cannot answer that if you do not know what data the backtest actually ran on.

Explicit handling of absence. A gap is modelled as a gap. The backtest knows the difference between "the market was closed", "we have no data for this day" and "the value was zero". Collapsing those three into one number is how you get a backtest that is precisely wrong.

A reference dataset for known-hard cases. We keep a small, carefully checked set of instruments and periods that include the nasty stuff — a delisting, a split, a known data revision. Every change to the data pipeline runs against it. If the pipeline ever starts handling a known-hard case differently, we find out immediately.

The pitfall that matters most

The dangerous failure here is not the backtest that crashes. It is the backtest that runs cleanly and gives you a great-looking result built on survivorship bias and a couple of bad adjustments. It looks like a discovery. It feels like a discovery. It is a data artifact, and the only thing standing between you and trading on it is whether you validated the inputs.

What we would tell you

If you are building anything quantitative on free or cheap data:

  • Validate before you backtest. The data layer needs its own checks, separate from the strategy.
  • Assume survivorship bias is present until you have specifically confirmed it is not.
  • Track provenance. "Which data did this run use" must be an answerable question.
  • Keep a hard-cases reference set and run every pipeline change against it.

Free data is fine. We use it daily. But it is an input that needs to earn trust, not one you get to assume is correct. The backtest will never tell you the data was bad. It will just be wrong, beautifully.

QuantBacktestingData QualityWeatherstoneFintech

Working on something similar?

We have probably hit the exact scaling or architectural bottlenecks you are navigating. Book a call and we'll walk through the technical trade-offs.