Skip to content
GetProfitable
Search

Drawdown, sample size and overfitting

Lesson 20 · about 12 min

Three questions decide whether a backtest means anything: how bad did it get, how many trades is the result based on, and how many knobs did you turn to get it. The tester answers the first directly, the second with a number most people skip past, and the third not at all.

Max drawdown

Max drawdown is the largest peak-to-trough decline in equity, shown in currency and as a percent in the Performance Summary and as a curve in the Overview. It is the number that will decide whether you keep trading the system, because you will live through it in real time without knowing whether it is the drawdown that ends or the one that keeps going.

Three things to do with it:

  • Scale it by sizing. Drawdown is roughly proportional to risk per trade. If 1% risk produced a 15% drawdown, 2% risk produces about 30%. Choose risk so that the historical drawdown, times a safety factor of 1.5 to 2, is one you can survive financially and psychologically. Future drawdowns are usually deeper than past ones.
  • Read its length as well as depth. The Overview drawdown curve shows how long equity stayed under water. Twelve months of no new highs breaks more traders than a sharp 20% dip.
  • Compare with net profit. Net profit divided by max drawdown (sometimes called the recovery factor) is a crude but useful ratio. Below 2 over several years means the system spends most of its life recovering.

Sample size

The Performance Summary's "Total closed trades" is the denominator of every other number. With 20 trades, a 60% win rate has a confidence interval of roughly 38% to 79%. With 100 trades, about 50% to 70%. With 400, about 55% to 65%. The standard error of a proportion is √(p(1−p)/n), and for an average trade it is the standard deviation of trade results divided by √n. Either way, halving the uncertainty takes four times the trades.

Rules of thumb that hold up:

  • Below 30 trades, the result is an anecdote.
  • 100 trades is the minimum for a first opinion.
  • 300 or more before sizing up.
  • More trades on more symbols beat more trades on one symbol; a rule that works on one contract and nowhere else is describing that contract's past, not a market behaviour.

A strategy that produces 12 trades a year on a daily chart needs a decade of data for a first opinion, and a decade covers only a handful of regimes. That is not a reason to switch to a 1-minute chart, where costs dominate; it is a reason to test across many symbols or to accept that daily-timeframe conclusions are weaker.

Overfitting

Overfitting is choosing parameters that fit the past better than they will fit the future. It is not a mistake you make on purpose; it is what happens whenever you look at results and adjust. The tester makes it effortless: change the EMA length, click, better; change the ATR multiple, click, better. After twenty clicks you have a curve that describes the historical data and a set of numbers that will not survive next month.

Signs you have done it:

  • Many inputs (more than four or five that meaningfully change results).
  • Results that fall apart when a parameter moves one step. If length 21 is great and 19 and 23 are poor, 21 is noise. Robust parameters sit on a plateau of similar results, not a spike.
  • Filters added to remove specific losing trades ("no trades on Fridays in December").
  • Great results on one symbol and one timeframe only.
  • A win rate or profit factor that is much higher than anything published for that style.

In-sample and out-of-sample

The standard defence is to develop on one period and check on another you did not look at:

//@version=6
strategy("IS/OOS demo", overlay=true, initial_capital=10000,
     default_qty_type=strategy.percent_of_equity, default_qty_value=10,
     commission_type=strategy.commission.percent, commission_value=0.05, slippage=1)
splitTime = input.time(timestamp("2023-01-01T00:00:00"), "In-sample ends")
mode = input.string("In-sample", "Test window", options=["In-sample", "Out-of-sample", "All"])
inSample  = time < splitTime
allowed = mode == "All" or (mode == "In-sample" ? inSample : not inSample)

fast = ta.ema(close, 9)
slow = ta.ema(close, 21)
if ta.crossover(fast, slow) and allowed
    strategy.entry("Long", strategy.long)
if ta.crossunder(fast, slow)
    strategy.close("Long")
bgcolor(inSample ? na : color.new(color.blue, 95))

Tune everything with "In-sample". When you are done, switch to "Out-of-sample" once and read the result. If it holds up (similar expectancy and profit factor, not identical), you have evidence. If it does not, you have learned something and you do not get to tune on the out-of-sample period and try again; that just makes it in-sample too.

A two-thirds / one-third split is common. Walk-forward testing extends the idea by rolling the split through time, which is beyond what the tester does natively but can be approximated by moving the split date and recording each result.

Parameter sensitivity by hand

The tester has no built-in optimiser in Pine, which is a feature. Test sensitivity manually: run the strategy at each of five values around your chosen parameter and write down the profit factor. If the numbers are 1.4, 1.5, 1.6, 1.5, 1.4, you are on a plateau. If they are 0.9, 1.1, 1.9, 1.0, 0.8, you are on a spike, and the true value is closer to the neighbours than the peak.

Key idea: Max drawdown scales with risk per trade and will be deeper than history shows; below 100 trades nothing is established; every parameter you tune on a period fits that period, so keep a period you never tune on.

Try it: Take a strategy with two inputs. Record the profit factor at five values of each on the in-sample window. Pick the plateau value, not the peak. Then run the out-of-sample window once and write down whether the expectancy sign held.

Recap

  • Max drawdown is proportional to risk per trade; size so that 1.5 to 2 times the historical drawdown is survivable, and read its duration.
  • Sample size sets the uncertainty; halving it takes four times the trades; 100 is a first opinion, 300 before sizing up.
  • Overfitting is the default outcome of tuning; robust parameters sit on plateaus and work across symbols.
  • Split data into in-sample and out-of-sample with input.time, tune only on the first, check the second once.
  • Test sensitivity manually across neighbouring parameter values.

See it drawn

Original diagrams for the ideas on this page. Illustrative, not real market data.

An equity curve and its drawdownAn account balance rising over a year, falling from a peak to a trough, then climbing back to the old peak.ACCOUNT EQUITY$20k$12k$8k024681012TIME (MONTHS)PEAK $16,000TROUGH $12,000DRAWDOWN−25%RECOVERY
Equity curve and drawdown. An account balance plotted month by month. The fall from the $16,000 peak to the $12,000 trough is a 25% drawdown, and the shaded area lasts until the balance climbs back to the old peak.
The spread of outcomes behind an expectancyA histogram of forty trades: a tall block of small losses on the left, a low spread of larger wins on the right, and a line marking the average outcome.NUMBER OF TRADES051024 LOSSES, AVG −$20016 WINS, AVG +$600EXPECTANCY +$120−$400−$200$0+$200+$400+$600+$800PROFIT OR LOSS PER TRADEexpectancy = (40% × $600) − (60% × $200) = +$120 per trade
Expectancy: the average trade. Forty trades sorted by outcome: 24 small losses and 16 larger wins. Weighting each side by how often it happens gives the average result per trade, marked here by the dashed line at +$120.
How a position size is worked outAccount size, risk per trade and stop distance feed into one box giving the number of shares.ACCOUNT SIZE$25,000your capitalRISK PER TRADE1%of the accountSTOP DISTANCE$0.50entry to stopPOSITION SIZE500 sharesrisk budget: $25,000 × 1% = $250position size: $250 ÷ $0.50 = 500 shares
Working out a position size. Three numbers decide how big a trade is: the account, the share of it put at risk, and the distance from entry to stop. One percent of $25,000 is a $250 budget, and a $0.50 stop divides into that 500 times.