Stop Calling the First Significant Day a Win

Daily peeking at A/B test dashboards inflates the nominal 5% false-positive rate to 27.7% when checked every day for 30 days. A seeded Python simulation demonstrates this inflation across different peeking frequencies (2 looks → 8.3%, 5 looks → 14%, 10 looks → 19.1%, 30 looks → 27.7%). Even when a real effect exists, early stopping exaggerates the measured lift — a true 10% lift appears as 12.7% when stopped at first significance. Three corrective approaches are benchmarked: fixed-sample no-peeking (5.1% FPR, 97.9% power, 30 days), Pocock group-sequential boundary with z=2.73 (4.9% FPR, 93.3% power, ~11 days), and always-valid p-values via mSPRT (1.5% FPR, 87.5% power, ~14 days). The key recommendation is to declare the stopping rule before the test starts and choose a method that matches how the team actually monitors experiments.

Source: https://towardsdatascience.com/stop-calling-the-first-significant-day-a-win. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.