Hypothesis first
Lesson 1 · about 9 min
A backtest is an experiment. Like any experiment, it only means something if you decided what you were testing before you looked at the results. Most bad backtests are not bad because of the software or the data. They are bad because the trader started clicking through indicators, found something that looked good, and then wrote the "idea" down afterwards.
What a hypothesis looks like
A trading hypothesis has three parts: a claim about behaviour, a reason that behaviour might exist, and a way it could be shown false.
| Part | Weak version | Usable version |
|---|---|---|
| Claim | "Breakouts work" | "In liquid US stocks, a close above the 20-day high is followed by a higher close 10 days later more often than not" |
| Reason | "Everyone knows that" | "Stops from short sellers and momentum buyers cluster above recent highs, so a break pulls in forced buying" |
| Falsifier | (none) | "If the 10-day forward return after such a close is not different from the unconditional 10-day return, the claim is wrong" |
The reason matters more than it looks. An edge with no plausible mechanism is far more likely to be noise, and when it stops working you will have no way to tell whether the mechanism broke or you were simply unlucky. "Who is on the other side, and why are they willing to lose to me?" is the question that turns a pattern into a hypothesis.
Write it down before you test
Open a plain text file and record, dated:
- The claim, in one sentence.
- The mechanism you think is behind it.
- The market, timeframe and date range you will test on.
- What result would make you abandon the idea.
- Roughly what you expect to see (win rate, average R, trades per month).
Point 5 is the one traders skip and the one that catches you later. If you predicted "about 45% winners, 1.5R average, 8 trades a month" and the test shows 71% winners and 4R average, that is not a reason to celebrate. It is a reason to look for a bug, a look-ahead leak, or an accidental filter that only works on this particular sample.
Key idea: Decide what you are testing, why it should work, and what would prove it wrong, before you run a single bar. A pattern found by searching and explained afterwards is not a hypothesis; it is a story fitted to noise.
Exploration is allowed, but label it
None of this means you cannot poke around. Looking at charts, running scans, and playing with parameters is how ideas arrive. The rule is to label that work as exploration and to treat anything you find there as a hypothesis to be tested on data you have not yet used, not as a result.
A simple habit: keep two folders. "Exploration" holds everything you found by looking. "Tests" holds only experiments that were written down first. Only results from the second folder count when deciding whether to trade something.
A worked example
Suppose you notice on a chart that EUR/USD often reverses after the London open. Here is the same observation turned into a testable hypothesis.
| Field | Entry |
|---|---|
| Claim | The direction of the first 30 minutes after 08:00 London is reversed more often than continued over the following two hours |
| Mechanism | The initial move is Asian-session positions being squared by European desks; once flat, the flow reverses |
| Data | EUR/USD 5-minute bars, 2018 to 2022, London time |
| Falsifier | If continuation happens 50% of the time or more, the claim fails |
| Expectation | Around 55% reversals, small average move, maybe 0.2R after spread |
Notice that the expectation is modest. A modest expectation that is met is worth more than a huge one that is not.
Try it: Take the last trading idea that excited you. Fill in the five-line template above. If you cannot write a mechanism, or cannot describe a result that would make you drop the idea, stop there. That is the finding.
What the backtest is not for
A backtest cannot generate an idea for you. Running an optimiser across every indicator and every parameter will always produce something that looks wonderful on past data; the later modules of this course explain exactly why. The backtest's job is to reject bad hypotheses cheaply, so that only ideas with a mechanism and a track record on unseen data reach your live account.
Recap
- A hypothesis is a claim, a mechanism, and a condition that would prove it wrong.
- Write the claim, data range, falsifier and expected numbers down before testing.
- Results that are far better than you expected are a warning, not a win.
- Keep exploration and formal tests in separate folders; only tests count.
- The backtest's job is to reject ideas cheaply, not to invent them.
See it drawn
Original diagrams for the ideas on this page. Illustrative, not real market data.