Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Panel Experiments

Abstract

Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. Beyond aiding managerial decision making, experiments mitigate risk by limiting the proportion of customers exposed to innovations. Because many experiments are conducted sequentially over time, an emerging strategy to further derisk the process is to allow managers to “peek” at the results as new data become available and stop the test if the results are statistically significant. The class of statistical methods that allow managers to peek and still provide valid inference are often called anytime valid because they maintain proper uniform type 1 error guarantees. In this paper, we extend existing anytime-valid approaches to accommodate the more complex yet standard settings in time series, switchback, and panel experiments. To achieve this, we leverage the design-based approach to focus on assumption-light and managerially relevant finite-sample estimands defined on the study participants as a direct measure of the risks incurred by companies. As a special case, our (asymptotic) results also provide a robust method for achieving always-valid inference in A/B tests. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that using our confidence sequence, harmful experiments could be stopped after observing only a handful of units; for instance, our method would have stopped a 30,000-person Netflix experiment after the first 100 people. This paper was accepted by J. George Shanthikumar, data science. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.08235 .