Backtesting
In backtesting you apply rules to old price data and see what would have come out. The test is only as good as the care you take to avoid the typical mistakes.
Learning objective: After this lesson you can check a backtest result for typical mistakes instead of believing it blindly.
The idea sounds tempting. You take your rules, run them over the last ten years and see what would have come out. In practice almost every backtest is too optimistic.
Common Mistakes
The most common mistakes are always the same. You use data that didn't exist yet at the time of the decision. You only test with companies that still exist today and overlook the ones that went bust. Or you forget fees, the spread and taxes.
The Mistake You Make Yourself
The sneakiest mistake is you. You already know the past. If your first test looks bad and you tweak the settings until it looks good, you haven't tested anything. You've fitted the rule to the past.
How to Test More Cleanly
In practice that means: fix the rules completely before you look at the data. Test on one period and check the result on another that you haven't touched before. And be generous with costs, not stingy.
Two mistakes distort backtests especially badly. The first is looking into the future. Quarterly results, for example, are only published weeks after the quarter ends. If your test acts as though you knew them on the last day of the quarter, it's trading on knowledge nobody had. The second is survivorship bias: data sets that only contain companies that still exist today. Wirecard then never appears, and your strategy never bought it. Both make results look much better than they were.
Restated Figures
With figures from annual reports there's something else. Companies sometimes correct published figures afterwards. Today's databases then contain the corrected values and not what was known at the time. Reliable tests need data that record when each figure became known.
Separating Development and Testing
Splitting the period into a part for developing and a part for checking is necessary but not enough. If you keep checking on the same test period and change your rules in between, at some point you fit them to that period too. What helps is ongoing testing on ever new data and an honest count of how many variants you've tried. If you tested a hundred variants, the best one is very likely just luck.
Summary
- Almost every backtest is too optimistic.
- Fix the rules before you look at the data.
- A test period is no longer independent once you use it several times.
Did you get it?
What is survivorship bias in a backtest?
The data set only contains companies that still exist today. The failed ones are missing.
Why are figures from annual reports a problem?
Because they get corrected afterwards and today's databases don't show what was known at the time.
Why isn't a test period split off once enough?
Because it loses its independence as soon as you use it several times and change your rules in between.
What others often ask about this
Why does my strategy work in the backtest but not live?
Usually it's fitted too closely to the old data. Add costs, spread and slippage that are missing from the test, and sometimes a mistake where the test uses information that wasn't available at the time.
How do I test more honestly?
Test on data you didn't look at while developing, include realistic costs and be suspicious of very smooth results.
Check your understanding
Your task
Work out how many dollars 100 trades of $5,000 each cost per year at 0.2 percent per trade. Then decide how many years of your data you will leave untouched as a validation period before you run the test once, and write down your rules completely before you look at the data.
Sources and further reading
- The widely cited paper on the probability of backtest overfitting, showing why standard statistical safeguards against overfitting are unreliable for trading strategies. View source ↗
Related
- Automated tradingStage 5
- Portfolio theoryStage 5
- Hedging with optionsStage 5
- Matching questions for this stageQuestions