A two-phase method for estimating treatment effects of a policy rolled out to a subset of a very large population (hundreds of millions of units) without a classical A/B test. Inspired by synthetic control, it estimates counterfactual outcomes for treated units from unaffected control units: first nearest-neighbor matching on covariates selects similar controls (mitigating interpolation bias), then high-dimensional supervised learning predicts counterfactuals. The paper documents a regularization (machine-learning) bias arising from bias-variance trade-offs in the ML estimators and proposes de-biasing corrections, validated across six large-scale experiments.
Shima Nassiri (Lyft), Mohsen Bayati (Amazon/Stanford) and Joe Cooprider (Amazon) study post-launch evaluation of policies (treatments) as an alternative to costly A/B tests in very-high-dimensional s…