Skip to content
GetProfitable
Search
Dictionary

Research log

A written record of every idea tested, with parameters, results and the decision taken. It is the only way to know how many tests your final result survived.

Without a log you cannot apply a bonferroni-correction, estimate a false-discovery-rate, or answer the single most important question about any backtest: how many other versions did you try before this one.

Record the hypothesis before running the test, then the result and the decision. Writing the hypothesis first is uncomfortable precisely because it prevents retrofitting an explanation to whatever the data produced, which is p-hacking in its most natural form.

A log also compounds. Ideas that failed for a specific reason, wrong instrument, wrong horizon, too costly, are often revivable later under different conditions, and a year of notes is a far better starting point than a year of memories.

Related: multiple-testing, data-snooping, p-hacking, bonferroni-correction

Educational only, not advice. Spotted an error? Post in Site Feedback.