The bias that arises when you test many ideas on one dataset and report only the winner, whose performance is inflated by luck.
If you test 100 independent strategies that have no edge at all, roughly 5 will look significant at the 5% level. Report only those and you have a research paper made entirely of noise.
The inflation is quantifiable. The expected maximum of 100 draws from a standard normal is about 2.5 standard deviations. So with 100 trials on zero-edge strategies, the best one will typically show a sharpe-ratio around 2.5 standard errors above zero, which on ten years of daily data is roughly 0.8.
Snooping also happens across people. Every published momentum variant was found by someone searching the same price history, so the collective trial count is enormous even if your own is small. See multiple-testing and deflated-sharpe-ratio.
Original diagrams for the ideas on this page. Illustrative, not real market data.
Expectancy: the average trade. Forty trades sorted by outcome: 24 small losses and 16 larger wins. Weighting each side by how often it happens gives the average result per trade, marked here by the dashed line at +$120.
Educational only, not advice. Spotted an error? Post in Site Feedback.