Jupyter, pandas, numpy and matplotlib

Four tools do almost all of the work in this course. Jupyter is where you explore; pandas holds tables of bars; numpy does the arithmetic underneath; matplotlib draws the pictures. This lesson is a fast tour of the parts you will actually use, with the trading-specific habits that save time later.

Jupyter for exploring, files for keeping

Start a notebook server from inside your activated venv:

jupyter notebook

A browser tab opens. Create a new Python 3 notebook and you can run code cell by cell, seeing the output of each. This is ideal for looking at data: load it, print the head, plot it, try an idea, plot again.

It is a poor place to keep a strategy. Cells can be run out of order, so a notebook that "works" may only work because you ran cell 12 before cell 4 an hour ago. The rule this course follows: explore in notebooks, then move anything you want to keep into a .py file that runs top to bottom. Module 1, lesson 3 covers the layout.

pandas: the DataFrame is a table of bars

A Series is one column with an index. A DataFrame is several columns sharing an index. For price data, the index is time and each row is a bar.

import numpy as np
import pandas as pd

rng = np.random.default_rng(1)
idx = pd.bdate_range("2024-01-01", periods=8)
df = pd.DataFrame(
    {
        "close": 100 + rng.normal(0, 1, 8).cumsum(),
        "volume": rng.integers(1_000, 5_000, 8),
    },
    index=idx,
)
print(df)
print(df.dtypes)
print(df.index.dtype)

The operations you will use constantly:

df["close"].pct_change()          # simple return, bar over bar
df["close"].shift(1)              # yesterday's close on today's row
df["close"].rolling(3).mean()     # 3-bar moving average
df.loc["2024-01-03":"2024-01-05"] # slice by date label
df.iloc[-1]                       # last row by position
df[df["volume"] > 3000]           # boolean filter

shift(1) deserves special attention now, because it is the tool that stops you from cheating. df["close"].shift(1) on the row for 5 January holds the close of 4 January. When a rule needs "the information I had at the time," you will reach for shift again and again. Module 5 makes this precise.

Two habits worth adopting from day one. First, keep column names lowercase and consistent: open, high, low, close, volume. Every function in this course assumes those names. Second, keep the index sorted and unique; call df.sort_index() after loading and check df.index.is_unique.

numpy: vectors instead of loops

pandas is built on numpy arrays. When you write df["close"] * 2, numpy multiplies every element in compiled code. Writing the same thing as a Python for loop is often a hundred times slower and much easier to get wrong.

closes = df["close"].to_numpy()
log_returns = np.diff(np.log(closes))
print(log_returns.round(4))
print("mean", log_returns.mean().round(5), "std", log_returns.std(ddof=1).round(5))

ddof=1 gives the sample standard deviation, which is what you want for return statistics; numpy's default is ddof=0, pandas' default is ddof=1. That difference will show up as a small disagreement between two Sharpe ratio implementations if you forget it.

Useful numpy functions: np.log, np.exp, np.cumsum, np.cumprod, np.where(condition, a, b), np.sign, np.maximum.accumulate (running maximum, used for drawdowns). You will meet all of them.

matplotlib: a chart in three lines

import matplotlib.pyplot as plt

ax = df["close"].plot(title="Synthetic close", figsize=(10, 4))
ax.set_ylabel("price")
plt.tight_layout()
plt.savefig("close.png", dpi=120)
plt.show()

pandas objects have a .plot() method that returns a matplotlib axis, so most charts start from the DataFrame rather than from plt. For an equity curve with a drawdown panel:

equity = (1 + df["close"].pct_change().fillna(0)).cumprod()
drawdown = equity / equity.cummax() - 1

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 6), sharex=True,
                               gridspec_kw={"height_ratios": [3, 1]})
equity.plot(ax=ax1, title="Equity")
drawdown.plot(ax=ax2, title="Drawdown", color="red")
ax2.fill_between(drawdown.index, drawdown, 0, color="red", alpha=0.3)
plt.tight_layout()
plt.show()

You will use this two-panel layout for every backtest, so it is worth saving as a function.

Key idea: Think in whole columns. Anything you would write as "for each bar, do X" can almost always be written as one pandas expression, and the column version is faster, shorter and harder to get subtly wrong.

A note on floating point

0.1 + 0.2 == 0.3 is False in Python, as in every language using IEEE floats. When you compare prices or check that positions sum to zero, use a tolerance: abs(a - b) < 1e-9 or np.isclose(a, b). When you compute quantities to send to a broker, round explicitly to the instrument's lot size. Getting this wrong in a backtest produces a wrong number; getting it wrong live produces a rejected order.

Try it: In a notebook, build the 8-row DataFrame above, then compute df["close"].pct_change() and df["close"] / df["close"].shift(1) - 1. Confirm they are identical with np.allclose, ignoring the first NaN. Then explain, in one sentence, why the first row is NaN.

Recap

• Explore in Jupyter; keep anything permanent in .py files that run top to bottom.
• A DataFrame with a sorted datetime index and lowercase open/high/low/close/volume columns is the standard shape for bars.
• shift, pct_change, rolling and boolean filters cover most day-to-day work.
• Prefer whole-column numpy operations to Python loops.
• Compare floats with a tolerance and round order quantities explicitly.