Skip to content
GetProfitable
Search
Dictionary

Reality check test

A statistical test that asks whether the best strategy out of many tested beats a benchmark by more than the search itself would produce by chance.

Ordinary significance tests judge one strategy. The reality check judges the best of N, which is a different and much harder question. It works by bootstrapping the whole set of strategy returns jointly, recording the maximum performance in each bootstrap draw, and comparing your actual maximum to that distribution.

Because the strategies are resampled together, their correlations are respected. Testing 500 variants of one moving-average rule counts for much less than 500 genuinely different ideas, which is correct and which bonferroni-correction gets wrong.

Expect deflating answers. Sets of a few hundred technical rules tested on index data frequently fail the reality check even when the best rule shows a Sharpe above 1.

Related: data-snooping, multiple-testing, bootstrap, deflated-sharpe-ratio

Educational only, not advice. Spotted an error? Post in Site Feedback.