cicyt UNIZAR
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Methodology

Title: Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls

Abstract: Online controlled experiments are the primary tool for measuring the causal impact of product changes in digital businesses. It is increasingly common for digital products and services to interact with customers in a personalised way. Using online controlled experiments to optimise personalised interaction strategies is challenging because the usual assumption of statistically equivalent user groups is violated. Additionally, challenges are introduced by users qualifying for strategies based on dynamic, stochastic attributes. Traditional A/B tests can salvage statistical equivalence by pre-allocating users to control and exposed groups, but this dilutes the experimental metrics and reduces the test power. We present a stacked incrementality test framework that addresses problems with running online experiments for personalised user strategies. We derive bounds that show that our framework is superior to the best simple A/B test given enough users and that this condition is easily met for large scale online experiments. In addition, we provide a test power calculator and describe a selection of pitfalls and lessons learnt from our experience using it.
Comments: Under review
Subjects: Methodology (stat.ME); Discrete Mathematics (cs.DM); Applications (stat.AP)
Cite as: arXiv:1803.06258 [stat.ME]
  (or arXiv:1803.06258v1 [stat.ME] for this version)

Submission history

From: C.H. Bryan Liu [view email]
[v1] Fri, 16 Mar 2018 14:59:06 GMT (702kb,D)