Skip to content
GetProfitable
Search

Kill switches, idempotent orders and state

Lesson 27 · about 14 min

The failures that hurt in live trading are not wrong signals. They are the program running twice and buying twice, the program crashing halfway and leaving an order it forgot about, and the program continuing to trade through a day when a human would have stopped. Three mechanisms cover those: idempotent orders, persisted state with reconciliation, and kill switches. None is more than thirty lines, and none is optional.

Idempotent orders

An operation is idempotent if doing it twice has the same effect as doing it once. Order submission must be idempotent, because cron can fire twice, a network timeout can hide a successful submission, and you will one day run the script by hand while it is also scheduled.

The mechanism is the client_order_id from lesson 1: a deterministic id built from everything that identifies the intent, so the same intent always produces the same id and the broker rejects the second copy.

import hashlib


def client_order_id(strategy: str, symbol: str, run_date: str, side: str) -> str:
    """Deterministic id: same intent on the same day -> same id -> broker rejects duplicates."""
    raw = f"{strategy}|{symbol}|{run_date}|{side}"
    return f"{strategy[:8]}-{hashlib.sha1(raw.encode()).hexdigest()[:16]}"


print(client_order_id("crossover", "SYNTH", "2024-06-03", "buy"))
print(client_order_id("crossover", "SYNTH", "2024-06-03", "buy"))       # identical
print(client_order_id("crossover", "SYNTH", "2024-06-04", "buy"))       # different day, different id

Quantity is not part of the id. If the second run computes a slightly different quantity because cash changed, you still want it rejected; the intent ("go long SYNTH today") is the same. Most brokers enforce uniqueness of client order ids per account; the paper broker from lesson 1 does too. Before relying on it, confirm in the broker's docs and test it against the paper endpoint.

Persisted state

The program exits after every run, so anything it needs to remember must be on disk. Keep it small and human-readable.

# src/state.py
import json
from dataclasses import dataclass, asdict, replace
from pathlib import Path


@dataclass(frozen=True)
class RunState:
    last_run_date: str = ""
    last_client_order_id: str = ""
    expected_qty: float = 0.0
    consecutive_failures: int = 0
    start_of_day_equity: float = 0.0


def load_state(path: str = "state/run_state.json") -> RunState:
    p = Path(path)
    if not p.exists():
        return RunState()
    return RunState(**json.loads(p.read_text()))


def save_state(state: RunState, path: str = "state/run_state.json") -> None:
    p = Path(path)
    p.parent.mkdir(parents=True, exist_ok=True)
    tmp = p.with_suffix(".tmp")
    tmp.write_text(json.dumps(asdict(state), indent=2))
    tmp.replace(p)                       # atomic on POSIX: never a half-written state file

Write to a temporary file and rename it. If the process dies mid-write, the old state survives intact instead of leaving a truncated JSON file that crashes the next run.

expected_qty is what the program believes the position is after its last order. That belief is checked against reality at the start of every run.

Reconciliation

import logging
from src.alerts import send_alert            # lesson 2
from src.broker import run_once              # lesson 1

log = logging.getLogger("live")


def reconcile(broker, symbol: str, state: RunState, tolerance: float = 0.0) -> bool:
    positions = broker.get_positions()
    actual = positions[symbol].qty if symbol in positions else 0.0
    if abs(actual - state.expected_qty) > tolerance:
        log.error("reconcile MISMATCH symbol=%s expected=%.4f actual=%.4f", symbol, state.expected_qty, actual)
        return False
    log.info("reconcile ok symbol=%s qty=%.4f", symbol, actual)
    return True

If the broker says you hold 950 shares and the state file says 0, something happened that the program does not know about: a partial fill, a manual trade, a run that crashed after submitting. The correct response is to stop and alert, not to "fix" it by trading to the expected quantity. A human looks, understands, updates the state file, and re-enables. The one thing an automated system must never do is take a large corrective action on the basis of a discrepancy it does not understand.

Key idea: Orders carry a deterministic client id so a repeated submission is harmless. State lives in an atomically-written file. Every run begins by checking that the broker's reality matches the program's belief, and halts on any mismatch.

Kill switches

Three, layered, each cheap:

1. A file. If state/HALT exists, the program logs, alerts and exits before doing anything. Creating a file is something you can do from a phone over SSH in ten seconds, with no code change and no broker login.

2. A daily loss limit. Compare equity now with equity at the start of the day; if the drop exceeds the limit from your risk plan, flatten (or at least stop opening), write the HALT file, and alert.

3. A failure counter. If the last N runs failed, stop trying. A program that has failed three days running is not going to succeed on the fourth without a human.

from pathlib import Path


def check_kill_switches(broker, state: RunState, symbol: str, max_daily_loss: float = 0.03,
                        max_failures: int = 3, halt_path: str = "state/HALT") -> str | None:
    """Return a reason to halt, or None if trading may proceed."""
    if Path(halt_path).exists():
        return "HALT file present"
    if state.consecutive_failures >= max_failures:
        return f"{state.consecutive_failures} consecutive failures"
    positions = broker.get_positions()
    last = float(broker.get_bars(symbol, 1)["close"].iloc[-1])
    equity = broker.get_cash() + sum(p.qty * last for p in positions.values())
    if state.start_of_day_equity > 0:
        loss = 1 - equity / state.start_of_day_equity
        if loss > max_daily_loss:
            Path(halt_path).parent.mkdir(exist_ok=True)
            Path(halt_path).write_text(f"daily loss {loss:.2%} exceeded {max_daily_loss:.2%}")
            return f"daily loss {loss:.2%}"
    return None

The loss limit writes the HALT file itself, so once tripped it stays tripped until a human deletes the file. That is deliberate: the system does not decide when it is safe to resume.

The run, with everything wired in

def guarded_run(broker, symbol: str, run_date: str, state: RunState) -> RunState:
    reason = check_kill_switches(broker, state, symbol)
    if reason:
        log.error("halted reason=%s", reason)
        send_alert("error", f"{run_date} halted: {reason}")
        return state
    if not reconcile(broker, symbol, state):
        send_alert("error", f"{run_date} reconcile mismatch; HALT written")
        Path("state/HALT").write_text("reconcile mismatch")
        return state
    try:
        ack = run_once(broker, symbol, run_date)
        held = broker.get_positions().get(symbol)
        return replace(state, last_run_date=run_date,
                       last_client_order_id=ack.client_order_id if ack else state.last_client_order_id,
                       expected_qty=held.qty if held else 0.0, consecutive_failures=0)
    except Exception:
        log.exception("run failed")
        return replace(state, consecutive_failures=state.consecutive_failures + 1)

guarded_run returns a new state (never mutates the old one), and main saves it. Note that expected_qty is taken from the broker after the order, not from the requested quantity: if the fill was partial, the state reflects the truth.

Monitoring: the heartbeat

Alerts tell you about events. A heartbeat tells you about the absence of events.

import time


def write_heartbeat(path: str = "state/heartbeat") -> None:
    Path(path).parent.mkdir(exist_ok=True)
    Path(path).write_text(str(int(time.time())))


def heartbeat_age_seconds(path: str = "state/heartbeat") -> float:
    p = Path(path)
    return float("inf") if not p.exists() else time.time() - int(p.read_text())

The run writes the heartbeat on success. A second, separate cron job (or a free external "dead man's switch" service that expects a ping) checks its age every hour and alerts if it exceeds a day. The two jobs fail independently, which is the point: the monitor is not the thing being monitored.

Try it: Using the paper broker, write tests for each mechanism: submit the same order twice and assert one duplicate; write a state file, change the paper broker's position by hand, and assert reconcile returns False; set start_of_day_equity high and assert the loss limit writes the HALT file; and assert guarded_run returns without submitting when the HALT file exists. Four tests, four failure modes that can no longer surprise you.

Recap

  • Deterministic client_order_id from strategy, symbol, date and side makes repeated submissions harmless.
  • State is a small frozen dataclass, written atomically via temp file and rename.
  • Reconcile expected against actual position at the start of every run; halt on mismatch, never auto-correct.
  • Kill switches: a HALT file, a daily loss limit that writes it, and a consecutive-failure counter.
  • A heartbeat checked by an independent job detects the runs that never happened.

See it drawn

Original diagrams for the ideas on this page. Illustrative, not real market data.

A fast and a slow moving average crossingA jagged price line with two smoother average lines through it; the fast average dips below the slow one on the left and cuts back above it in the middle, where a circle marks the crossing.pricefast averageslow averagefast crosses belowfast crosses abovethe slow averageAverages of recent closes; the fast one reacts sooner than the slow one.
Fast and slow moving averages crossing. A moving average is the average of the last few closing prices, redrawn each period. An average over fewer periods turns sooner than one over many, so the two lines cross whenever the recent pace of the market changes.

Finished this module? Take the module quiz.