Skip to content
GetProfitable
Search
Dictionary

False discovery rate

The expected share of your accepted discoveries that are wrong. A gentler target than eliminating all false positives, and usually the right one for strategy search.

Controlling the family-wise error rate with bonferroni-correction asks for almost no false positives at all. Controlling FDR instead accepts that some of what you keep will be junk, and caps the proportion, say at 10%.

The Benjamini-Hochberg procedure is mechanical: sort your m p-values ascending, find the largest k where p(k) <= (k/m) x q, and keep the first k. With 100 tested strategies, q = 0.10, and a sorted list where the 6th smallest p-value is 0.0055 and the 7th is 0.02, you would keep six because 6/100 x 0.10 = 0.006 > 0.0055 while 7/100 x 0.10 = 0.007 < 0.02.

For a research pipeline that will allocate small amounts to many strategies and cut the failures, tolerating a known proportion of duds is rational. For a single all-in deployment it is not.

Related: bonferroni-correction, multiple-testing, p-value, data-snooping

Educational only, not advice. Spotted an error? Post in Site Feedback.