Skip to content
GetProfitable
Search
Dictionary

Data quality check

Automated assertions run over every dataset before it reaches a model: gaps, duplicates, impossible values, stale rows, and suspicious jumps.

A minimal set: high is at least the max of open and close, low is at most the min, volume is non-negative, timestamps are strictly increasing with no duplicates, no more than N consecutive identical closes, and any absolute return above a threshold is flagged for review rather than dropped.

Run these as tests that fail loudly. Silent coercion, filling a missing price with the previous one, is how a stale feed becomes a flat line that a mean-reversion system happily trades against for a week.

Worked example: flag any daily move greater than 8 times the trailing 60-day standard deviation. On a normal universe this fires a handful of times a year, and each firing is either a genuine event worth knowing about or a bad-tick worth fixing.

Related: bad-tick, missing-data, outlier, data-vendor

Educational only, not advice. Spotted an error? Post in Site Feedback.