Concluding a strategy has an edge when it does not. The expensive error in trading, because you fund it.
A false positive costs real money twice: the capital lost trading a coin flip, and the months of attention not spent on something that worked. Every backtest you keep because it looked good is a candidate type I error.
The rate of these is controlled by your significance-level, but only per test. Search 500 parameter combinations at the 5% level and you should expect roughly 25 to look significant even if none has any edge. This is why the count of things you tried is as important a number as the p-value, and why serious research logs every variant.
Practical defence: reserve a holdout-set you look at once, size the first live allocation as though the edge might be zero, and require a out-of-sample period before the position grows.
Original diagrams for the ideas on this page. Illustrative, not real market data.
Expectancy: the average trade. Forty trades sorted by outcome: 24 small losses and 16 larger wins. Weighting each side by how often it happens gives the average result per trade, marked here by the dashed line at +$120.
Educational only, not advice. Spotted an error? Post in Site Feedback.