> There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?"

Relevant keywords here include 'trial sequential analysis', 'adaptive trials', & 'multi-armed bandits'.