Overfitting
Search long enough, and you'll find regularities in any dataset that are pure chance. The more variants tested, the more certain you are to find one that shines in hindsight.
Test a thousand random rules against historical data, and some will look outstanding. Not because they work, but because with a thousand attempts, something's always going to stand out.
That's exactly what happens when optimizing parameters. You try periods, thresholds, and filters until the curve looks good. The result then describes the past perfectly and the future not at all.
A warning sign is a too-smooth result curve. Real strategies have bad stretches. If a backtest has none, it was usually adjusted until they disappeared.
The best protection is restraint. Few rules, few parameters, round values instead of exactly optimized ones. A strategy with two parameters that looks mediocre is usually worth more than one with ten that shines.
With k independent attempts, the probability of getting at least one apparently significant result approaches one. Correction methods account for the number of attempts, for instance through adjusted significance levels or a Sharpe ratio adjusted for the number of trials. Without stating how many variants were tested, a result can't be evaluated.
The effective number of attempts is often larger than stated, since choosing the universe, period, and data source already involves degrees of freedom. Even the decision to continue an investigation at all depends on interim results, which is an implicit form of selection.
Countermeasures include limiting the number of parameters, checking stability around the chosen parameters, rolling forward-testing, and requiring an economic rationale before analyzing the data. The stability check is especially revealing: a result that collapses with a minor parameter change is very likely just chance.
Summary
- With enough attempts, something always shines in hindsight.
- A too-smooth result curve is a warning sign, not a seal of quality.
- A result that collapses with a small parameter change is chance.
Did you get it?
Why does the number of variants tested matter so much?
Because with many attempts, apparently significant results turn up by chance alone.
What does a very smooth result curve suggest?
Usually that it was adjusted until the bad stretches disappeared.
What does a stability check test?
Whether the result holds up under a minor change to the parameters.
Related
- Chart patternsStage 3
- BacktestingStage 4
- What a company is worthStage 3