Grid search and the overfitting trap

Once a backtest is a function of its parameters, the temptation is to call it for every combination and keep the best. That is a grid search, it takes five lines, and it is the most reliable way ever devised to produce a beautiful backtest of a strategy that does not work. This lesson runs one, shows what the result looks like on data with no edge at all, and gives you the two habits that keep grid searches useful: look at neighbourhoods, not cells, and count your trials.

Running a grid

import itertools
import numpy as np
import pandas as pd
from src.data import synthetic_ohlcv
from src.indicators import sma

TRADING_DAYS = 252


def crossover_returns(close: pd.Series, fast: int, slow: int, cost_bps: float = 5.0) -> pd.Series:
    signal = (sma(close, fast) > sma(close, slow)).astype(float)
    position = signal.shift(1).fillna(0.0)
    turnover = position.diff().abs().fillna(0.0)
    return (position * close.pct_change()).fillna(0.0) - turnover * cost_bps / 10_000.0


def sharpe(r: pd.Series) -> float:
    sd = r.std(ddof=1)
    return float(r.mean() / sd * np.sqrt(TRADING_DAYS)) if sd > 0 else float("nan")


def grid_search(close: pd.Series, fasts: list[int], slows: list[int]) -> pd.DataFrame:
    rows = [
        {"fast": f, "slow": s, "sharpe": sharpe(crossover_returns(close, f, s))}
        for f, s in itertools.product(fasts, slows) if f < s
    ]
    return pd.DataFrame(rows)


bars = synthetic_ohlcv(1500, seed=42)
fasts = [5, 10, 15, 20, 30, 40, 50]
slows = [20, 30, 50, 75, 100, 150, 200]
grid = grid_search(bars["close"], fasts, slows)
print(grid.sort_values("sharpe", ascending=False).head(5))
print(f"{len(grid)} combinations tested")

The top row is the "optimal" parameter set. On a synthetic random walk with no structure whatsoever, it will have a respectable Sharpe ratio, because among forty-odd noisy numbers the largest is always respectable. That is the whole trap in one sentence.

The heat map

Look at the surface, not the summit.

import matplotlib.pyplot as plt

surface = grid.pivot(index="fast", columns="slow", values="sharpe")
fig, ax = plt.subplots(figsize=(8, 5))
im = ax.imshow(surface.to_numpy(), cmap="RdYlGn", aspect="auto", origin="lower")
ax.set_xticks(range(len(surface.columns)), surface.columns)
ax.set_yticks(range(len(surface.index)), surface.index)
ax.set_xlabel("slow"); ax.set_ylabel("fast"); ax.set_title("Sharpe by parameter")
plt.colorbar(im, ax=ax)
plt.tight_layout()
plt.show()

Two kinds of picture. In one, there is a broad region where most cells are similar and mildly positive, with the best cell somewhere inside it. In the other, the best cell is a bright spot surrounded by cells that are much worse. The first is a plateau; the second is a spike. A plateau means the rule's behaviour is not sensitive to the exact numbers, which is what a real, if modest, effect looks like. A spike means the numbers happen to fit the noise of this sample.

Measure it instead of eyeballing it:

def neighbourhood_score(grid: pd.DataFrame, fast: int, slow: int) -> float:
    """Mean Sharpe of the cell and its immediate neighbours in the grid."""
    fi = sorted(grid["fast"].unique()); si = sorted(grid["slow"].unique())
    f_near = [f for f in fi if abs(fi.index(f) - fi.index(fast)) <= 1]
    s_near = [s for s in si if abs(si.index(s) - si.index(slow)) <= 1]
    cells = grid[grid["fast"].isin(f_near) & grid["slow"].isin(s_near)]
    return float(cells["sharpe"].mean())


best = grid.loc[grid["sharpe"].idxmax()]
print(f"best cell ({int(best.fast)}, {int(best.slow)}): sharpe {best.sharpe:.2f}, "
      f"neighbourhood {neighbourhood_score(grid, int(best.fast), int(best.slow)):.2f}")

Pick parameters by neighbourhood score, not by cell. You will give up a little in-sample performance and gain a great deal of out-of-sample survival.

What noise looks like

The most useful experiment in this module: run the identical grid on many random walks and record the best Sharpe from each.

best_sharpes = []
for seed in range(30):
    close = synthetic_ohlcv(1500, seed=seed, drift=0.0)["close"]
    g = grid_search(close, fasts, slows)
    best_sharpes.append(g["sharpe"].max())
best_sharpes = pd.Series(best_sharpes)
print(f"best in-sample Sharpe on pure noise: median {best_sharpes.median():.2f}, "
      f"90th pct {best_sharpes.quantile(0.9):.2f}, max {best_sharpes.max():.2f}")

With zero drift, no strategy has any edge. Yet the best cell of the grid regularly reports a Sharpe near or above 1, and sometimes well above. Now you have a baseline: a grid-search-optimal Sharpe on your real data has to clear this number by a wide margin before it means anything, and the more cells you searched, the higher the number.

Key idea: The best cell of a grid is the maximum of many noisy estimates and is biased upwards by construction. Judge parameters by the neighbourhood, compare the best in-sample result to what the same search produces on pure noise, and record how many combinations you tried.

Counting trials honestly

Every combination you evaluate is a trial, and so is every rule variant, every instrument, every cost assumption, and every "let me just try one more thing". The expected maximum Sharpe of n independent zero-edge strategies grows roughly like √(2 ln n) times the standard error of a single Sharpe estimate. With a few hundred trials the expected best is comfortably above 1 on five years of daily data. The correction (the "deflated Sharpe ratio" of Bailey and López de Prado) is beyond this course, but the input to it is simple: keep a count.

TRIAL_LOG = "trials.csv"


def log_trial(name: str, params: dict, sharpe_value: float) -> None:
    row = pd.DataFrame([{"name": name, "params": str(params), "sharpe": sharpe_value,
                         "when": pd.Timestamp.now(tz="UTC")}])
    row.to_csv(TRIAL_LOG, mode="a", header=False, index=False)

Call it from inside grid_search. A trials log that says "412 combinations across 3 strategy families" next to a headline Sharpe of 1.4 tells a reader (including future you) exactly how much to discount it.

Fewer parameters, wider steps

The cheapest defence against overfitting is to have less to fit. A rule with two parameters on a coarse grid (7 × 7 = 49 cells) has far less room to fit noise than one with five parameters on a fine grid (10⁵ cells). Prefer rules with few parameters; step them in ratios (10, 20, 40) rather than units (10, 11, 12); and treat any parameter that has to be tuned to the third significant figure as evidence that it is fitting noise.

Try it: Rerun the noise experiment with a finer grid (fasts and slows in steps of 2 from 4 to 200, keeping fast < slow). Record the median best Sharpe. Then rerun with the coarse grid. The difference between the two medians is the price of the extra cells, paid in false confidence.

Recap

• A grid search is itertools.product plus a backtest function; the best cell is biased upwards by the number of cells.
• Judge parameters by the neighbourhood mean, not the peak; plateaus generalise, spikes do not.
• Run the same search on drift-free synthetic data to see what the best cell looks like when there is no edge.
• Log every trial; the count is the input to any honest correction.
• Fewer parameters on coarser, ratio-spaced grids leave less room to fit noise.