Purged cross-validation and honest reporting

Walk-forward uses each stretch of history as a test segment exactly once and always tests on data that is later than the training data. Cross-validation, the standard tool in machine learning, tests on every segment by rotating which one is held out, which gives more test data per unit of history. On time series it also leaks, unless you take specific precautions. This lesson explains the leak, implements the purge-and-embargo fix, and closes the module with the reporting standard the whole course has been building towards.

Why ordinary k-fold leaks on prices

Split five years into five folds, train on four, test on one, rotate. Two problems.

First, the training data now includes the future relative to the test fold. A 200-day moving average computed at the start of fold 3 uses bars from fold 2, which is fine, but the parameters were chosen using folds 4 and 5, which happened later. For pure parameter selection on a stationary effect this is arguably acceptable; for anything with drift, it is not, and you cannot tell which case you are in.

Second, and more subtly, the labels overlap. Suppose you evaluate a signal by the 20-day forward return of each bar. The bar at the end of fold 2 has a forward return that extends 20 days into fold 3. Train on fold 2, test on fold 3, and the last 20 training labels contain fold 3's prices. The model has seen the beginning of the test set. This leak is invisible in the code and it inflates every score.

Purging and embargo

The fix has two parts:

• Purge: remove from the training set any observation whose label period overlaps the test fold.
• Embargo: additionally remove a small buffer of training observations after the test fold, because serial correlation lets information leak backwards too.

import numpy as np
import pandas as pd


def purged_kfold_indices(n: int, n_splits: int, label_horizon: int, embargo: int):
    """Yield (train_idx, test_idx) with purge and embargo. Indices are positional."""
    fold_edges = np.linspace(0, n, n_splits + 1, dtype=int)
    all_idx = np.arange(n)
    for k in range(n_splits):
        test_start, test_end = fold_edges[k], fold_edges[k + 1]        # [start, end)
        test_idx = all_idx[test_start:test_end]
        # purge: training rows whose label window [i, i + horizon) touches the test fold
        purge_start = max(0, test_start - label_horizon)
        # embargo: rows immediately after the test fold
        embargo_end = min(n, test_end + embargo)
        keep = (all_idx < purge_start) | (all_idx >= embargo_end)
        yield all_idx[keep], test_idx


for train_idx, test_idx in purged_kfold_indices(n=1000, n_splits=5, label_horizon=20, embargo=10):
    before = train_idx[train_idx < test_idx[0]]
    gap = test_idx[0] - before.max() - 1 if len(before) else 0
    print(f"test [{test_idx[0]:>4}, {test_idx[-1]:>4}]  train rows: {len(train_idx):>4}  purged gap before test: {gap:>3}")

Each fold now has a 20-bar hole before the test segment and a 10-bar hole after it, and the training set is everything else. Fewer training rows, no leak.

Using it to evaluate a signal

A common use is to ask whether a feature has any predictive relationship with forward returns, judged out of fold.

from src.data import synthetic_ohlcv
from src.indicators import rsi

bars = synthetic_ohlcv(2000, seed=42)
horizon = 10
feature = rsi(bars["close"], 14)
label = bars["close"].shift(-horizon) / bars["close"] - 1      # forward return: analysis only
df = pd.DataFrame({"x": feature, "y": label}).dropna()

scores = []
for train_idx, test_idx in purged_kfold_indices(len(df), n_splits=5, label_horizon=horizon, embargo=5):
    train, test = df.iloc[train_idx], df.iloc[test_idx]
    threshold = train["x"].quantile(0.2)                         # "oversold" learned on train only
    oos_signal_ret = test.loc[test["x"] < threshold, "y"]
    scores.append(float(oos_signal_ret.mean()) if len(oos_signal_ret) else np.nan)

print(f"mean OOS 10-bar return after low RSI, per fold: {np.round(scores, 4)}")
print(f"average: {np.nanmean(scores):.4f}")

On the synthetic walk the per-fold numbers straddle zero. On real data, a signal worth pursuing shows the same sign in most folds; a signal that is enormous in one fold and absent in the others is one regime, not an edge.

The full "combinatorial purged cross-validation" of López de Prado goes further: instead of k test folds it forms every combination of several test folds, producing many backtest paths and a distribution of results rather than one number. The idea is the same as the Monte Carlo of Module 6, applied to fold assignment rather than trade order. The implementation is a few dozen more lines and a good next project once the simple purged version is familiar.

Key idea: On time series, cross-validation must purge training labels that overlap the test fold and embargo a buffer after it. Judge a signal by the consistency of its sign across folds, not by the average alone.

The reporting standard

A backtest report that a sceptical stranger would accept contains, at minimum:

1. Data: source, date range, adjustment method, how survivorship was handled, and the data_report numbers.
2. Execution assumptions: fill price (close or next open), cost in basis points, and the result at double that cost.
3. The rule in one paragraph, with every parameter and its value.
4. Search history: the number of parameter combinations, rule variants, instruments and cost assumptions tried, from the trials log. Not the ones that worked; all of them.
5. Out-of-sample results only as the headline: walk-forward or purged-CV numbers. In-sample numbers appear only next to their out-of-sample counterparts.
6. The window-by-window log: parameter choices per window and the fraction of windows that were positive.
7. Variance: bootstrap 5th-percentile drawdown and the t-stat of trade expectancy.
8. Known omissions: financing, impact, taxes, and anything else you did not model.

def report_card(oos_returns: pd.Series, trials: int, cost_bps: float, windows_positive: int, windows: int) -> str:
    from math import sqrt
    sd = oos_returns.std(ddof=1)
    s = oos_returns.mean() / sd * sqrt(252) if sd > 0 else float("nan")
    eq = (1 + oos_returns).cumprod()
    mdd = (eq / eq.cummax() - 1).min()
    return (f"OOS Sharpe {s:.2f} | OOS max DD {mdd:.1%} | cost {cost_bps:.0f} bps | "
            f"trials {trials} | positive windows {windows_positive}/{windows}")


print(report_card(pd.Series(np.random.default_rng(0).normal(0.0002, 0.01, 1000)), 49, 5.0, 4, 8))

The one-line card is what goes at the top; the eight items are what backs it up. If a number in the card cannot be traced to one of the eight, it does not go in the card.

The discipline behind the standard

Everything in this module reduces to one habit: decide the procedure before seeing the result, and report the procedure along with the result. A grid of 49 cells, a walk-forward with 3-year training and 1-year testing, rolling, parameters chosen by neighbourhood score, costs at 5 and 10 bps, decided in advance and written in config.py, is a test. The same steps chosen after looking at what worked is a story. Both produce an equity curve; only one of them tells you anything.

Try it: Take the walk-forward from lesson 2 and produce all eight items of the reporting standard for it on the synthetic data, including the trials count and the bootstrap drawdown percentile from Module 6. Then write the one-line report card. It will describe a strategy with no edge, honestly, which is the skill.

Recap

• Ordinary k-fold leaks on time series through training-in-the-future and through labels that overlap the test fold.
• Purge training rows whose label window touches the test fold and embargo a buffer after it.
• Judge signals by sign consistency across folds; combinatorial purged CV extends this to a distribution of paths.
• The report: data, execution assumptions, rule, full search history, out-of-sample headline, window log, variance, omissions.
• Decide the procedure before seeing results; a procedure chosen afterwards is a story, not a test.