Anytime-Valid Multi-Objective Online Experimentation with Contrast Certificates
Abstract
Large-scale search and recommendation systems routinely run online experiments for ranking models, feed policies, user-interface treatments, and ad/recommendation blending strategies. These launches are rarely single-metric decisions: teams often need to improve click-through rate, conversion, dwell time, retention, or revenue while controlling negative feedback and latency. We propose AV-MO-SOPTIMA++, an anytime-valid framework for multi-objective online experimentation under adaptive traffic allocation and optional stopping. Instead of relying only on marginal arm-objective confidence sequences, AV-MO-SOPTIMA++ uses prospective two-sample contrast confidence sequences for decision-relevant arm pairs; these contrasts may be pre-activated for a fixed decision mode or activated online when new bottleneck constraints appear. The method derives Pareto and guardrail certificates from these contrasts and propagates certified dominance through a dominance graph. The framework further uses decision-specific stopping rules for full Pareto, relaxed Pareto, and guardrail launch decisions, together with a bottleneck-aware allocation policy that prioritizes comparisons currently blocking a valid decision. Experiments show that the anytime-valid controller maintains calibration under optional stopping, exposes false certificates from nominal multi-objective peeking, and supports safe guardrail launch decisions with fewer samples than fixed-horizon A/B or full-Pareto PSI in a launch benchmark. Component ablations show that contrast certificates reduce guardrail stopping time and bottleneck allocation sharply reduces bad-arm exposure. In full Pareto identification, contrast variants reduce samples on separated instances, while near-boundary instances remain challenging for all validity-first methods.