Distance is time. Light travels roughly 30 centimetres per nanosecond in vacuum and slower in fibre, so a server 50 kilometres away is at least several hundred microseconds behind one in the building.
Exchanges sell colocation as a regulated product with equalised cable lengths, so every cabinet in the hall has the same physical path to the engine. That is the fairness argument: within the hall nobody is closer, and outside it everybody is further.
Example: a cabinet plus cross-connects and power might run $10,000 to $25,000 a month per venue, before feeds and hardware. A strategy earning $0.0008 per share needs roughly 20 to 30 million shares a month just to cover one venue's colocation bill.
Related: latency, direct-feed, high-frequency-trading, matching-engine