Project layout and reproducibility

A trading project grows. What starts as one notebook becomes a data loader, a handful of indicators, three strategy variants, a backtester and, eventually, a script that talks to a broker. If those live in one file, every change risks breaking something you cannot see. This lesson sets up the layout the rest of the course builds into, and the habits that let you trust a number you computed six months ago.

The layout

algo-course/
  .venv/                 # never committed
  .gitignore
  requirements.txt
  config.py              # constants: paths, symbols, fees
  data/
    raw/                 # downloaded files, treated as read-only
    clean/               # parquet files produced by our code
  notebooks/             # exploration only
  src/
    __init__.py
    data.py              # load, validate, synthetic generators
    indicators.py        # SMA, EMA, RSI, ATR
    strategies.py        # rules → signals
    backtest.py          # signals → returns / trades
    metrics.py           # CAGR, Sharpe, drawdown
    broker.py            # adapter to a broker API (Module 9)
  tests/
    test_indicators.py
  run_backtest.py        # entry point

The split is by responsibility, not by size. data.py knows nothing about strategies; strategies.py knows nothing about brokers. When a backtest number looks wrong, you can test each layer alone.

Create the skeleton:

mkdir -p data/raw data/clean notebooks src tests
touch src/__init__.py config.py run_backtest.py
printf ".venv/\ndata/\n__pycache__/\n*.png\n.ipynb_checkpoints/\n" > .gitignore

data/ is ignored because raw files are large and often not yours to redistribute; keep the script that downloads them instead.

Constants live in one place

# config.py
from pathlib import Path

ROOT = Path(__file__).resolve().parent
RAW_DIR = ROOT / "data" / "raw"
CLEAN_DIR = ROOT / "data" / "clean"

SYMBOLS = ["SPY_SYNTH", "QQQ_SYNTH"]   # placeholders; see Module 2
BAR_COLUMNS = ["open", "high", "low", "close", "volume"]

COMMISSION_PER_TRADE = 0.0005   # 5 bps of notional, round trip charged per side
SLIPPAGE_BPS = 2                # basis points per fill
TRADING_DAYS = 252
SEED = 42

Everything that could plausibly change (a fee, a symbol, a path) is here, not scattered as literals in five files. When you later ask "what commission did I assume in the March backtest?", the answer is in git history, not in your memory.

The data module starts with a synthetic generator

Real data comes in Module 2. Until then, and for every test afterwards, you want a generator that produces plausible OHLCV bars deterministically.

# src/data.py
import numpy as np
import pandas as pd

BAR_COLUMNS = ["open", "high", "low", "close", "volume"]


def synthetic_ohlcv(n: int = 1000, seed: int = 42, start: str = "2020-01-01",
                    drift: float = 0.0003, vol: float = 0.012) -> pd.DataFrame:
    """Deterministic daily OHLCV bars from a geometric random walk."""
    rng = np.random.default_rng(seed)
    rets = rng.normal(drift, vol, n)
    close = 100.0 * np.exp(np.cumsum(rets))
    open_ = np.concatenate(([100.0], close[:-1])) * (1 + rng.normal(0, 0.002, n))
    wick_up = rng.uniform(0, 0.008, n)
    wick_dn = rng.uniform(0, 0.008, n)
    high = np.maximum(open_, close) * (1 + wick_up)
    low = np.minimum(open_, close) * (1 - wick_dn)
    volume = rng.integers(500_000, 5_000_000, n).astype(float)
    index = pd.bdate_range(start, periods=n, name="date")
    df = pd.DataFrame(
        {"open": open_, "high": high, "low": low, "close": close, "volume": volume},
        index=index,
    )
    return df


def validate_bars(df: pd.DataFrame) -> pd.DataFrame:
    """Raise on the data problems that silently corrupt backtests."""
    missing = [c for c in BAR_COLUMNS if c not in df.columns]
    if missing:
        raise ValueError(f"missing columns: {missing}")
    if not df.index.is_monotonic_increasing:
        raise ValueError("index is not sorted")
    if not df.index.is_unique:
        raise ValueError("index has duplicate timestamps")
    bad = (df["high"] < df[["open", "close"]].max(axis=1)) | (
        df["low"] > df[["open", "close"]].min(axis=1)
    )
    if bad.any():
        raise ValueError(f"{int(bad.sum())} bars have high/low inconsistent with open/close")
    if (df[BAR_COLUMNS] <= 0).any().any():
        raise ValueError("non-positive prices or volume")
    return df

validate_bars is deliberately strict. A single duplicated timestamp doubles one day's return in a naive backtest; a high below the close means a data vendor error that will make any stop-loss logic nonsense. Fail loudly at load time rather than discovering it as a suspiciously good result.

A test that runs in one second

# tests/test_indicators.py
import numpy as np
from src.data import synthetic_ohlcv, validate_bars


def test_synthetic_is_deterministic():
    a = synthetic_ohlcv(100, seed=1)
    b = synthetic_ohlcv(100, seed=1)
    assert np.allclose(a["close"], b["close"])


def test_synthetic_passes_validation():
    validate_bars(synthetic_ohlcv(200))

Run with pip install pytest then pytest -q from the project root. Tests like this are cheap and they catch the day you accidentally change the generator and every backtest shifts.

Key idea: Reproducibility is a property of the whole pipeline: pinned packages, a fixed seed, data files you can regenerate or re-download, and constants in one file. If any link is missing, the number on the screen is an anecdote.

The entry point

# run_backtest.py
from src.data import synthetic_ohlcv, validate_bars
import config

if __name__ == "__main__":
    bars = validate_bars(synthetic_ohlcv(seed=config.SEED))
    print(bars.tail(3))
    print(f"{len(bars)} bars from {bars.index[0].date()} to {bars.index[-1].date()}")

Run it with python run_backtest.py from the project root. As the course goes on, this file grows to: load data, compute indicators, generate signals, backtest, print metrics, save a chart. Each of those is one function call into src/.

Try it: Build the skeleton, paste in config.py, src/data.py, the test file and run_backtest.py. Run pytest -q and python run_backtest.py. Then deliberately break the data by adding df.iloc[5, df.columns.get_loc("high")] = 1.0 before validation and confirm validate_bars refuses it.

Recap

• Split the project by responsibility: data, indicators, strategies, backtest, metrics, broker.
• Constants (fees, symbols, seed, paths) live in config.py.
• A deterministic synthetic_ohlcv generator lets every lesson and test run without downloads.
• validate_bars fails loudly on sorted-index, duplicate, and high/low consistency problems.
• One-second tests protect you from silently changing the foundation.