Survivorship and look-ahead bias
Lesson 5 · about 11 min
Two data problems produce more fake edges than all the others combined. Survivorship bias means the data set only contains things that made it to today. Look-ahead bias means a rule is using information that was not available when the trade would have been placed. Both make a backtest look better than reality, and both are easy to introduce without noticing.
Survivorship bias
Take the current constituents of a large-cap index and backtest a "buy the dip" strategy over the past fifteen years. It will look excellent. Every one of those companies is, by construction, a company that survived and grew enough to be in the index today. The ones that dipped and kept dipping until they were delisted are not in your data, so your strategy never bought them.
| Universe | What is in it | Effect on a long-only test |
|---|---|---|
| Today's index members, tested on past data | Only survivors | Upward bias; dips "always" recovered |
| Index members as of each date (point-in-time) | Survivors and failures | Honest |
| All stocks including delisted, point-in-time | Everything | Honest, and includes the worst cases |
The effect is not small. Studies of US equity universes have found that survivorship-biased data can add several percentage points a year to a mean-reversion strategy's apparent return, and can flip a losing strategy to a winning one. It matters most for strategies that buy weakness, hold through drawdowns, or trade small caps, because those are the situations where companies vanish.
Futures and forex have a milder form: a currency pair or a contract you would have traded in 2008 may not be quoted anymore, and the choice of which instruments to test is usually made from today's list. Crypto has it in an extreme form. Backtesting "buy the top 20 coins by market cap" on today's top 20 excludes the dozens that were top 20 in a prior year and have since gone to near zero.
The fix is point-in-time data: a universe defined by what was known on each date. Where that is not available, at least be aware of the direction of the bias and discount long-side mean-reversion results accordingly.
Look-ahead bias
Look-ahead bias is any use of future information in a rule. It is rarely deliberate. Some common forms:
- Using the close of the bar you enter on. "If close > 20-day high, buy at the open." Fine. "If close > 20-day high, buy at the close" is only fine if you can actually place the order in the last second before the close, knowing the close. Most people cannot.
- Indicators computed on the whole series. A centred moving average, or any smoothing that uses bars on both sides, knows the future at every point.
- Repainting indicators. Some pivot, zigzag and fractal indicators redraw earlier points as new data arrives. Their historical plot shows only the version that was correct in hindsight.
- Fundamental data with the wrong date. Earnings for Q1 are known in late April or May, not on 31 March. If the data set stamps them at quarter end, a rule using them is trading on information that did not exist yet.
- Adjusted data used for level rules. A stop at "$50" on a dividend-adjusted series is a stop at a price that never printed. Next lesson.
- Daily high and low used within the bar. "Buy if the price touches the low of the day and rises 1%." On daily bars, you do not know the low of the day until the day is over.
Key idea: At every bar, ask: what did I actually know at the moment this order would have gone in? Any rule that needs information from later than that moment is look-ahead, and its results are fiction.
A worked look-ahead
A daily-bar strategy: enter long at the close when the close is above the 50-day average, exit at the close when it is below. On paper this is a clean trend rule. In a spreadsheet, the entry row uses the same close both for the signal and the fill.
| Day | Close | 50-day avg | Signal | Fill (as written) | Fill (honest) |
|---|---|---|---|---|---|
| 1 | 100.0 | 100.5 | none | ||
| 2 | 101.2 | 100.6 | long | 101.2 | next open, 101.6 |
| 3 | 102.5 | 100.7 | hold | ||
| 4 | 100.1 | 100.8 | exit | 100.1 | next open, 99.7 |
As written, the trade made 100.1 − 101.2 = −1.1. Honest fills give 99.7 − 101.6 = −1.9. On this trade the difference is 0.8 points. Across a few hundred trades, entering on a strong close and exiting on a weak close systematically captures a small overnight move that was never available. This bias alone can make a break-even system look profitable.
Checking for it
Two quick tests catch most look-ahead:
- Shift the fills one bar later. If the result collapses, the edge lived in the leak.
- Run the strategy bar by bar in a replay tool. If the signal you see on the replay differs from the historical plot, something is repainting.
Try it: Take a strategy of yours that enters on the close of the signal bar. Change it to enter at the next bar's open and rerun. Record the change in average R. If it fell by more than a quarter, most of what you had was the close-to-open leak.
Recap
- Survivorship bias: today's list of instruments excludes the ones that failed. It inflates long-side and mean-reversion results.
- Use point-in-time universes, or at least discount results on today's constituents.
- Look-ahead bias: any rule using information not available at order time.
- Common sources: fill on the signal close, centred or repainting indicators, misdated fundamentals, intra-bar highs and lows on bar data.
- Test by shifting fills one bar later and by replaying bar by bar.