Abstract
Online controlled experiments often struggle to measure small treatment effects in metrics characterized by high variance and long-tailed distributions. Conventional approaches of increasing sample size to mitigate statistical noise can require impractically large user populations or extended experiment durations. This disclosure describes a system that utilizes pre-experiment data as a covariate within a regression adjustment model to reduce variance in outcome metrics. The system incorporates a data processing pipeline to collect baseline user activity, a power analysis toolkit for prospective efficiency calculations, and an automated bias detection module to monitor group imbalances. By accounting for pre-existing behavioral patterns, the technology enhances statistical precision and power. This allows for the detection of smaller effects with reduced sample sizes, facilitating more efficient experimentation and data-driven decision-making.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Katz, Roy, "Covariate Adjustment for Variance Reduction in Online Controlled Experiments", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/9717
COinS