Multiple testing and the deflated Sharpe

If you flip a coin ten times, getting eight heads is unusual; it happens about 5% of the time. If a hundred people each flip a coin ten times, several of them will get eight heads, and one of them will probably get nine. The person with nine heads has not discovered a skill. This is the multiple-testing problem, and it is the mechanism behind almost every backtest that looked wonderful and then did nothing.

Why searching finds something

Every strategy variant you test is a coin flip against the noise in the data. Some fraction of pure-noise strategies will pass any threshold you set, and the more you test, the more certain it becomes that at least one passes.

If a single noise strategy has a 5% chance of clearing your bar, then after N independent tries:

probability at least one passes = 1 − 0.95^N

Twenty variations is a quiet evening's work. After that, finding a "significant" result is more likely than not even if nothing is there. And an optimiser running a parameter grid tests thousands.

The expected best result from noise

The multiple-testing effect can be estimated. For N independent trials of pure noise, the best Sharpe ratio you should expect to see is roughly:

expected maximum Sharpe ≈ standard error of Sharpe × √(2 × ln N)

The standard error of an annualised Sharpe computed on T years of daily data is approximately 1 ÷ √T for modest Sharpe values. So:

Read the table this way: on five years of data, after trying 100 variations, the best of them will show a Sharpe around 1.4 by luck alone. A backtest with a Sharpe of 1.4 that was the best of 100 tries on five years is exactly what noise produces. The same Sharpe found on the first try, or on 20 years of data, is much stronger evidence.

The deflated Sharpe ratio

Bailey and López de Prado formalised this in a 2014 paper introducing the deflated Sharpe ratio. The idea has three parts:

1. Estimate how many trials were effectively run (N), and how much the results varied across them.
2. Compute the Sharpe you would expect the best of N noise trials to produce (the table above is a simplified version of this).
3. Ask whether the observed Sharpe is significantly higher than that expected maximum, given the sample length and adjusting for non-normal returns (skewness and fat tails make the Sharpe estimate less reliable, and the full method accounts for that).

The output is a probability that the observed Sharpe exceeds the noise benchmark. Below about 95%, the strategy has not distinguished itself from the best of the things you tried.

A worked, simplified version:

An excess of 0.69 standard errors corresponds to roughly a 75% probability that the strategy is better than noise. That is not nothing, but it is well short of the 95% one would want. The same strategy after 20 variations instead of 60: expected max 0.45 × 2.45 = 1.10, excess 0.50, 1.1 standard errors, about 86%. On the first try: excess 1.6 ÷ 0.45 = 3.6 standard errors, and the result is convincing.

The exact numbers depend on assumptions the full method handles better, but the direction is the lesson: the same backtest is strong evidence if it was the first thing you tried and weak evidence if it was the sixtieth.

Key idea: The evidence in a backtest depends on how many other backtests you ran to find it. Count your trials honestly, including parameter grids and abandoned variants, and compare the result with the best that noise would produce over that many tries.

Counting trials honestly

The hard part is N. It includes:

• Every parameter combination an optimiser evaluated.
• Every variant you tried by hand and discarded.
• Every filter you added and removed.
• Every instrument you tested and rejected.
• Every strategy idea in your exploration folder that did not make it.

Traders systematically undercount. A conservative habit is to keep a log of every run, with a one-line description and the headline result. After a month the log will tell you your true N, and it is usually in the hundreds.

Trials are not fully independent (lookback 20 and lookback 21 give nearly the same trades), so the effective N is smaller than the raw count. Bailey and López de Prado suggest clustering similar trials. As a rough rule, count distinct ideas and clearly different parameter regions rather than every grid cell; but count all of them.

The practical response

1. Log every test.
2. Prefer few, well-motivated variants over grids.
3. When a result is found after many trials, demand a larger margin: a Sharpe well above the noise benchmark, out-of-sample and walk-forward confirmation, and a mechanism.
4. Treat any strategy found by brute-force search as unproven until it has survived a live incubation period, regardless of the backtest.

Try it: Count the number of variations you tested to arrive at your current best strategy, using the list above. Compute the expected maximum Sharpe from noise for that N and your data length. Compare with your observed Sharpe. If the gap is under one standard error, the backtest alone is not evidence of an edge.

Recap

• Testing many variants guarantees that some will look good by chance; 20 tries gives a 64% chance of a false pass at the 5% level.
• Expected maximum Sharpe from N noise trials ≈ SE × √(2 ln N); SE ≈ 1 ÷ √years.
• The deflated Sharpe (Bailey and López de Prado, 2014) asks whether the observed Sharpe beats that benchmark, adjusting for sample length and non-normality.
• Count trials honestly: grids, discarded variants, filters, instruments, abandoned ideas.
• A result found after many trials needs a much larger margin and live confirmation.