Skip to content
GetProfitable
Search
Dictionary

Data snooping

The bias that arises when you test many ideas on one dataset and report only the winner, whose performance is inflated by luck.

If you test 100 independent strategies that have no edge at all, roughly 5 will look significant at the 5% level. Report only those and you have a research paper made entirely of noise.

The inflation is quantifiable. The expected maximum of 100 draws from a standard normal is about 2.5 standard deviations. So with 100 trials on zero-edge strategies, the best one will typically show a sharpe-ratio around 2.5 standard errors above zero, which on ten years of daily data is roughly 0.8.

Snooping also happens across people. Every published momentum variant was found by someone searching the same price history, so the collective trial count is enormous even if your own is small. See multiple-testing and deflated-sharpe-ratio.

Related: multiple-testing, p-hacking, overfitting, deflated-sharpe-ratio

See it drawn

Original diagrams for the ideas on this page. Illustrative, not real market data.

The spread of outcomes behind an expectancyA histogram of forty trades: a tall block of small losses on the left, a low spread of larger wins on the right, and a line marking the average outcome.NUMBER OF TRADES051024 LOSSES, AVG −$20016 WINS, AVG +$600EXPECTANCY +$120−$400−$200$0+$200+$400+$600+$800PROFIT OR LOSS PER TRADEexpectancy = (40% × $600) − (60% × $200) = +$120 per trade
Expectancy: the average trade. Forty trades sorted by outcome: 24 small losses and 16 larger wins. Weighting each side by how often it happens gives the average result per trade, marked here by the dashed line at +$120.

Educational only, not advice. Spotted an error? Post in Site Feedback.