CSV from free sources and the yfinance caveats

Every backtest is only as good as its bars. Free daily data is plentiful and adequate for learning and for slow strategies; free intraday data is scarce, patchy, and usually not what a live system will see. This lesson covers loading a CSV correctly, what the popular yfinance package actually gives you, and the data problems you must check for before believing any result.

Where free daily data comes from

• Stooq publishes daily OHLCV for stocks, indices, forex and some futures as CSV downloads.
• Exchanges' own sites (Nasdaq, CME) publish some historical data and end-of-day settlements.
• Crypto exchanges (Binance, Coinbase, Kraken) expose public REST endpoints for candles at any interval with no API key; Binance also publishes bulk monthly CSVs.
• Broker APIs (next lesson) provide history to account holders, including paper accounts.
• yfinance is a Python package that scrapes Yahoo Finance. Convenient, unofficial, and the source of more subtle backtest bugs than any other free tool.

None of these should be a dependency of your code. Download once into data/raw/, then load from disk. Your backtest should never make a network call.

Loading a CSV properly

Suppose you downloaded a file that looks like this:

Date,Open,High,Low,Close,Volume
2024-01-02,470.12,473.67,468.90,472.65,63000000
2024-01-03,470.44,471.19,466.23,468.79,71000000

The naive pd.read_csv("file.csv") gives you a string Date column and capitalised headers. Do it once, correctly, in src/data.py:

import pandas as pd

BAR_COLUMNS = ["open", "high", "low", "close", "volume"]


def load_csv_bars(path: str) -> pd.DataFrame:
    """Load a daily OHLCV CSV into the standard bar format."""
    df = pd.read_csv(path)
    df.columns = [c.strip().lower().replace(" ", "_") for c in df.columns]
    date_col = "date" if "date" in df.columns else "timestamp"
    df[date_col] = pd.to_datetime(df[date_col], utc=True)
    df = df.set_index(date_col).sort_index()
    df.index.name = "date"
    if "adj_close" in df.columns:
        df = df.drop(columns=["adj_close"])
    df = df[BAR_COLUMNS].astype(float)
    df = df[~df.index.duplicated(keep="last")]
    return df

utc=True makes the index timezone-aware in UTC from the start; lesson 3 explains why that is non-negotiable. Dropping duplicates with keep="last" handles vendors that re-publish a corrected row. Converting all columns to float avoids the surprise where volume was parsed as integers and later division truncates.

To try this without downloading anything, write a synthetic CSV first:

from src.data import synthetic_ohlcv

bars = synthetic_ohlcv(300, seed=3)
bars.to_csv("data/raw/SYNTH_daily.csv", index_label="Date")
loaded = load_csv_bars("data/raw/SYNTH_daily.csv")
print(loaded.head(3))
print(loaded.index.tz)

The yfinance caveats

If you use yfinance, be aware of what you are getting.

Adjusted vs unadjusted prices. Yahoo's "Adj Close" is adjusted for dividends and splits; "Close" is adjusted for splits only in some versions of the library and for nothing in others, and the default of the auto_adjust flag has changed between releases. A strategy that computes stop distances from Close while returns come from Adj Close is comparing two different price series. Decide which you want, set auto_adjust explicitly, and record the choice in config.py.

Survivorship bias. You can only download tickers that exist today. A universe of "current S&P 500 members" tested back ten years excludes every company that went bankrupt or was removed. Any strategy that buys dips will look far better than it should.

Silent revisions. Yahoo's history for the same symbol and date can change between downloads. If you do not save the file you tested on, you cannot reproduce the test.

Intraday limits. Minute bars are available only for the last few weeks, hourly for about two years. That is enough to look, not enough to test.

Rate limits and breakage. It is a scraper of an unofficial endpoint and breaks periodically. Never put it in a live path.

The right way to use it: download once, save the raw CSV with the date of download in the filename, and never call it from the backtest.

Key idea: Data is an input you save and version, not a live dependency. Download, validate, store, then work from the stored file so the same code on the same file gives the same result forever.

Checks that catch most vendor problems

import numpy as np


def data_report(df: pd.DataFrame) -> dict:
    ret = df["close"].pct_change().dropna()
    gaps = df.index.to_series().diff().dt.days
    return {
        "bars": len(df),
        "start": str(df.index[0].date()),
        "end": str(df.index[-1].date()),
        "max_gap_days": int(gaps.max()),
        "zero_volume_bars": int((df["volume"] == 0).sum()),
        "abs_return_gt_20pct": int((ret.abs() > 0.20).sum()),
        "nan_cells": int(df.isna().sum().sum()),
    }


print(data_report(loaded))

A max_gap_days of 4 on daily stock data is normal (a long weekend). A gap of 40 means missing months. A 20%+ single-day move on a large index is almost always an unadjusted split. Zero-volume bars on a liquid stock are holidays that should not be there. Run this report on every file you load and look at the numbers before you look at any backtest.

Try it: Generate a synthetic CSV, then corrupt it by hand: duplicate one row, delete a week, and multiply one close by 2 (a fake split). Load it with load_csv_bars, run data_report and validate_bars, and confirm which checks catch which corruption. Note the one that nothing catches (the missing week only shows as a larger gap) and decide what threshold you would alert on.

Recap

• Free daily data is fine for learning; free intraday data is short and unreliable.
• Load CSVs through one function that normalises names, parses UTC dates, sorts, de-duplicates and casts to float.
• yfinance: adjustment flags vary, universes are survivorship-biased, history is revised, and it must never sit in a live path.
• Save the raw file you tested on; the backtest reads from disk, never the network.
• Run a data report (gaps, zero volume, extreme returns, NaNs) before trusting any result.