Backtesting
Backtesting applies rules to past data. The test is only as good as the care taken to rule out the usual sources of error.
The idea sounds compelling: take your rules, run them over the last ten years, and see what would have come out. In practice, almost every backtest is too optimistic.
The common mistakes are always the same. You use data that didn't exist yet at the decision point. You test only with companies that still exist today, missing the ones that went bankrupt. You forget fees, spread, and taxes.
The subtlest mistake is your own head. You know the past. If your first test looks bad and you tweak the parameters until it looks good, you haven't tested anything, you've fitted it.
In practice: define the rules completely before you look at the data. Test on one period, and verify on a different one you haven't touched before. And estimate costs generously, not tightly.
Systematic sources of error include look-ahead bias, when data gets used with a lag as if it had been instantly available, and survivorship bias in datasets that don't include delisted stocks. Both lead to substantial overstatement of results.
There's also the restatement problem with fundamental data: published figures get corrected after the fact, so today's datasets don't match what was actually known at the time. Robust studies require datasets timestamped to the actual point-in-time knowledge.
Splitting data into development and test periods is necessary but not sufficient, since repeated testing on the same test period erodes its independence. Techniques like rolling forward-testing and accounting for the number of variants tested address this. Without correcting for the number of attempts, a positive result carries little informational value.
Summary
- Almost every backtest is too optimistic.
- Fix the rules completely before looking at the data.
- A test period loses its independence once you reuse it multiple times.
Did you get it?
What is survivorship bias in a backtest?
The dataset only contains stocks that still exist today; the failed ones are missing.
Why are fundamental data problematic?
Because they get corrected after the fact, so today's datasets don't match what was known at the time.
Why isn't a once-separated test period enough?
Because it loses its independence once reused repeatedly.
Related
- The trading journalStage 2
- OverfittingStage 4
- Comparing brokersStage 1