Backtesting
Backtesting applies rules to past data. The test is only as good as the care taken to rule out the usual sources of error.
Learning objective: After this lesson, you can check a backtest result for typical sources of error instead of accepting it at face value.
The idea sounds compelling: take your rules, run them over the last ten years, and see what would have come out. In practice, almost every backtest is too optimistic.
The common mistakes are always the same. You use data that didn't exist yet at the decision point. You test only with companies that still exist today, missing the ones that went bankrupt. You forget fees, spread, and taxes.
The subtlest mistake is your own head. You know the past. If your first test looks bad and you tweak the parameters until it looks good, you haven't tested anything, you've fitted it.
In practice: define the rules completely before you look at the data. Test on one period, and verify on a different one you haven't touched before. And estimate costs generously, not tightly.
Systematic sources of error include look-ahead bias, when data gets used with a lag as if it had been instantly available, and survivorship bias in datasets that don't include delisted stocks. Both lead to substantial overstatement of results.
There's also the restatement problem with fundamental data: published figures get corrected after the fact, so today's datasets don't match what was actually known at the time. Robust studies require datasets timestamped to the actual point-in-time knowledge.
Splitting data into development and test periods is necessary but not sufficient, since repeated testing on the same test period erodes its independence. Techniques like rolling forward-testing and accounting for the number of variants tested address this. Without correcting for the number of attempts, a positive result carries little informational value.
Summary
- Almost every backtest is too optimistic.
- Fix the rules completely before looking at the data.
- A test period loses its independence once you reuse it multiple times.
Did you get it?
What is survivorship bias in a backtest?
The dataset only contains stocks that still exist today; the failed ones are missing.
Why are fundamental data problematic?
Because they get corrected after the fact, so today's datasets don't match what was known at the time.
Why isn't a once-separated test period enough?
Because it loses its independence once reused repeatedly.
What others often ask about this
Why does my strategy work in the backtest but not live?
Usually it's fitted too closely to the old data. Add costs, spread and slippage that are missing from the test, and sometimes a mistake where the test uses information that wasn't available at the time.
How do I test more honestly?
Test on data you didn't look at while developing, include realistic costs and be suspicious of very smooth results.
Check your understanding
Sources and further reading
- The widely cited paper on the probability of backtest overfitting, showing why standard statistical safeguards against overfitting are unreliable for trading strategies. View source ↗
Related
- Automated tradingStage 5
- Portfolio theoryStage 5
- Hedging with optionsStage 5
- Matching questions for this stageQuestions