Walk-forward analysis

The honest way to test a parameter-selection procedure is to run it exactly as you would in real life: choose parameters using only the past, trade them on the period that follows, then roll forward and repeat. Stitching the out-of-sample segments together gives an equity curve that never saw its own future. This is walk-forward analysis. It is the single most useful technique in this module, and it is about forty lines.

The procedure

1. Split history into consecutive windows. Each has a training segment (say 3 years) followed by a test segment (say 1 year).
2. In each window, run the grid search on the training segment only and pick parameters (by neighbourhood score, from the previous lesson, or simply by best cell if you want to see how badly that does).
3. Apply those parameters to the test segment and record the returns.
4. Slide the window forward by the test length and repeat.
5. Concatenate the test-segment returns. That is the walk-forward equity curve.

The training segments overlap; the test segments do not. Every test-segment return was generated by parameters chosen without any knowledge of it.

Implementation

import itertools
import numpy as np
import pandas as pd
from src.data import synthetic_ohlcv
from src.indicators import sma

TRADING_DAYS = 252


def crossover_returns(close: pd.Series, fast: int, slow: int, cost_bps: float = 5.0) -> pd.Series:
    signal = (sma(close, fast) > sma(close, slow)).astype(float)
    position = signal.shift(1).fillna(0.0)
    turnover = position.diff().abs().fillna(0.0)
    return (position * close.pct_change()).fillna(0.0) - turnover * cost_bps / 10_000.0


def sharpe(r: pd.Series) -> float:
    sd = r.std(ddof=1)
    return float(r.mean() / sd * np.sqrt(TRADING_DAYS)) if sd > 0 else float("nan")


def choose_params(close: pd.Series, fasts: list[int], slows: list[int]) -> tuple[int, int]:
    """Best cell on the training segment. Deliberately naive; see the Try it."""
    best, best_s = None, -np.inf
    for f, s in itertools.product(fasts, slows):
        if f >= s:
            continue
        val = sharpe(crossover_returns(close, f, s))
        if val > best_s:
            best, best_s = (f, s), val
    return best


def walk_forward(close: pd.Series, train_bars: int, test_bars: int,
                 fasts: list[int], slows: list[int]) -> tuple[pd.Series, pd.DataFrame]:
    oos_chunks, log = [], []
    start = 0
    while start + train_bars + test_bars <= len(close):
        train = close.iloc[start: start + train_bars]
        fast, slow = choose_params(train, fasts, slows)
        # include enough history before the test segment for the slow window to warm up
        warm_start = max(0, start + train_bars - slow)
        test_with_warmup = close.iloc[warm_start: start + train_bars + test_bars]
        r = crossover_returns(test_with_warmup, fast, slow).iloc[-test_bars:]
        oos_chunks.append(r)
        log.append({"test_start": r.index[0], "fast": fast, "slow": slow,
                    "is_sharpe": sharpe(crossover_returns(train, fast, slow)), "oos_sharpe": sharpe(r)})
        start += test_bars
    return pd.concat(oos_chunks), pd.DataFrame(log)


bars = synthetic_ohlcv(3000, seed=42)
fasts = [5, 10, 20, 30, 50]
slows = [30, 50, 100, 150, 200]
oos, log = walk_forward(bars["close"], train_bars=756, test_bars=252, fasts=fasts, slows=slows)
print(log.round(2))
print(f"\nwalk-forward OOS Sharpe: {sharpe(oos):.2f}   (in-sample mean: {log['is_sharpe'].mean():.2f})")

The warm-up handling matters. If you compute a 200-bar SMA on a test segment that starts cold, the first 200 bars have no signal, and a one-year test segment loses most of its trades. Feeding the test segment a slow-bar prefix from the training data and then keeping only the last test_bars returns fixes that without leaking anything, because the prefix is in the past.

Reading the log

The log frame is as informative as the equity curve:

• Parameter stability. If the chosen (fast, slow) jumps around wildly from window to window, the grid is fitting noise; a real effect tends to select similar parameters each time.
• In-sample versus out-of-sample. The is_sharpe column will be comfortably positive in every window, because it is the maximum of a search. The oos_sharpe column is the truth. The ratio of the OOS mean to the IS mean, sometimes called walk-forward efficiency, is typically well under 0.5 even for strategies that work. If it is near 1, be suspicious that the test segments leaked into training.
• Sign consistency. Count how many windows had positive OOS Sharpe. A strategy that is positive in 7 of 8 windows is more convincing than one with the same total return that made everything in one window.

print(f"windows with positive OOS: {(log['oos_sharpe'] > 0).sum()} of {len(log)}")
print(f"walk-forward efficiency: {log['oos_sharpe'].mean() / log['is_sharpe'].mean():.2f}")

On the synthetic random walk, expect OOS Sharpe near zero, efficiency near zero or negative, and parameters that jump. That is the correct answer for data with no edge, and seeing it once is worth a great deal: it is the picture of the honest test rejecting a strategy that a grid search would have accepted.

Key idea: Walk-forward analysis tests the whole procedure (search plus trade) rather than one parameter set. The stitched out-of-sample curve is the only equity curve from a searched strategy that deserves to be looked at, and the window-by-window log tells you whether the parameters were stable.

Choosing window lengths

Training needs enough trades for the search to be meaningful (for a daily crossover, three to five years). Testing needs enough bars to contain several trades (a year for slow rules, less for fast ones). Long training and short testing gives more windows and a smoother curve but each window's parameters are chosen from data that is mostly shared with the last window's. There is no right answer, only a stated one; the danger is trying several window lengths and reporting the best, which is a grid search on the validation procedure itself. Pick the lengths before you look at any result and log them.

Anchored versus rolling

The version above is rolling: each training segment is the same length and slides forward. An anchored walk-forward keeps the training start fixed and grows the segment each time, so later windows train on all of history. Anchored uses more data per window; rolling adapts faster to regime change. Try both once; report whichever you decided on in advance.

Try it: Replace choose_params with a version that picks the best neighbourhood score instead of the best cell (from lesson 1). Rerun the walk-forward on the same bars and compare OOS Sharpe and parameter stability. Then run both versions across ten seeds and average. The neighbourhood version should be less bad on noise and, on data with a real effect, less prone to collapsing.

Recap

• Choose parameters on a training segment, trade them on the following test segment, roll forward, stitch the test returns.
• Prepend a warm-up prefix from the training data so the test segment's indicators are valid from its first bar.
• Read the log: parameter stability, in-sample versus out-of-sample Sharpe, and the fraction of windows that were positive.
• Walk-forward efficiency well below 1 is normal; near 1 suggests leakage.
• Fix window lengths and rolling/anchored before looking at results, and do not search over them.