Skip to content
GetProfitable
Search
Dictionary

Train, validation, test split

Three separate data blocks: one to fit the model, one to choose between models, and one used once to estimate real performance.

The middle block is the one traders usually forget. If you pick your final model by comparing candidates on the same data you fitted them on, the comparison is contaminated. The validation block exists to make model selection itself honest.

A standard split for a ten-year daily dataset is six years train, two years validation, two years test, kept in time order so you never train on the future. Random shuffling is fine for photographs and wrong for time series.

Note that the validation block gets used many times, so its results are also optimistic by the end, just less so than the training block. Only the test block, used once, gives an unbiased estimate. See holdout-set.

Related: in-sample, out-of-sample, holdout-set, cross-validation

Educational only, not advice. Spotted an error? Post in Site Feedback.