...I thought I'd take advantage of those of you who work in web analytics and ask a question to those of you who might be involved in the A/B testing side of web analytics. :-)
\nI started reading about A/B testing in web analytics and the concept of stopping a test when significance is achieved came up. Due to my background in stats, I was puzzled with the whole idea of stopping an experiment when significance is observed, because my inclination would be to fix the sample size beforehand and just wait for the experiment to conclude. I also came across this blog, which re-enforced my skepticism towards early stopping.
\nSo, how do practitioners deals with this topic, and what are best practices to decide when/how often to peak to the data? Is it a function of time (i.e. every x hours), of the number of observations (i.e. every 1000 new observations). or something different?
", "messages": [ { "id": "58097", "postDate": "11/15/2014 01:32:32", "content": "...I thought I'd take advantage of those of you who work in web analytics and ask a question to those of you who might be involved in the A/B testing side of web analytics. :-)
\nI started reading about A/B testing in web analytics and the concept of stopping a test when significance is achieved came up. Due to my background in stats, I was puzzled with the whole idea of stopping an experiment when significance is observed, because my inclination would be to fix the sample size beforehand and just wait for the experiment to conclude. I also came across this blog, which re-enforced my skepticism towards early stopping.
\nSo, how do practitioners deals with this topic, and what are best practices to decide when/how often to peak to the data? Is it a function of time (i.e. every x hours), of the number of observations (i.e. every 1000 new observations). or something different?
", "rawMarkdown": "", "votes": null }, { "id": "58100", "postDate": "11/15/2014 03:52:19", "content": "I did online-advertising full time for a couple of years. I would set up a lots and lots of small tests, and kill the ones that were obvious statistical losers (X impressions, no clicks). Then I'd run them again, and repeat the analysis on the aggregated stats (XX impressions, < 10 clicks, or whatever). Only the strongest survived.
\nDid some some get prematurely culled? Of course. But you learn the hard way that it's far more cost effective to brutally cull the losers.
", "rawMarkdown": "", "votes": null }, { "id": "58101", "postDate": "11/15/2014 05:13:26", "content": "I worked at a major online site that runs many A/B tests. We shut off experiments that were clearly losing hand-over-fist in multiple metrics*, but otherwise we ran experiments for a period of time calculated in advance (4 weeks was common).** Stopping a test early just because significance seems to have been achieved is, generally speaking, statistically invalid. Often an experiment seemed to start out strong (random chance, imagine flipping heads many times at the beginning of a fair coin sequence) but then as more data was obtained, lost statistical significance.
Note that if you run lots of A/B tests, significance will be problematic. If you run 20 independent A/B experiments at p=0.05 with one metric each, on average one experimental result will be a type-I error and potentially lead to the wrong business conclusion. It helps to have multiple confirmation of principles among many different experiments (Marcello Truzzi: "Extraordinary claims require extraordinary proof") and it's also worth educating yourself about the family-wise error rate and logical fallacies such as the appeal to novelty. There are many A/B testing best-practices, such as running continuous A/A tests. Microsoft and Google have published some useful practical information about their A/B approaches.
I'm not sure what you mean by "peak the data," but if you mean to throttle experiment data beyond an arbitrary peak (such as the examples you gave), I would recommend against that. Throttling creates sampling errors and many other bad statistical effects.
(* by doing this, we would miss any tests that got an "unlucky" start and would have eventually dug out of that deficit and become positive. You're trading experimental recall for reduced risk of losses.)
(** We used whole weeks to average out day-of-week and time-of-day effects. But this doesn't catch longer seasonality patterns. An A/B test that performs well in summer might perform poorly in winter or vice-versa.)
Until recently I worked for a high volume online travel agent with a lot of A/B testing. Tests would sometimes be stopped early, when significance was established - but only when significance was well ahead of the target and high volumes of traffic had already gone through them, so we could be very confident they would have remained significant. It's a pragmatic trade-off between the cost/lost opportunity of running tests longer than needed, vs chance of drawing wrong conclusions.
\nIn addition to the very valid points above, you also have to think about the sort of traffic the test will have seen so far and whether there is reason to think it may be atypical. For example, if your website generates a lot of visits from a once-weekly email (where customers will generally be more qualified), and that email has not yet been sent, then it would arguably be foolish to stop a test before receiving this traffic no matter how significant it may currently be.
\nIn some sectors there may also be a case to run longer to ensure you are capturing return visits. If your customers generally require several "research" visits to your website before converting, then early results may represent the conversions from customers who started this process before the trial began and not be truly indicative of likely ongoing behaviour.
\nAnother point is that the longer you run a test the more you may be able to drill down to segments of interest. OK you might get topline significance before the test completes, but running for longer may allow deeper insights, eg allowing you to drill down results on different browsers/platforms, regionally etc.
", "rawMarkdown": "", "votes": null }, { "id": "58215", "postDate": "11/17/2014 17:56:51", "content": "Thanks guys! This was really helpful!
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 58100, "author_name": "inversion", "author_url": "", "post_date": "11/15/2014 03:52:19", "content": "I did online-advertising full time for a couple of years. I would set up a lots and lots of small tests, and kill the ones that were obvious statistical losers (X impressions, no clicks). Then I'd run them again, and repeat the analysis on the aggregated stats (XX impressions, < 10 clicks, or whatever). Only the strongest survived.
\nDid some some get prematurely culled? Of course. But you learn the hard way that it's far more cost effective to brutally cull the losers.
", "votes": null, "replies": [] }, { "id": 58101, "author_name": "brundage", "author_url": "", "post_date": "11/15/2014 05:13:26", "content": "I worked at a major online site that runs many A/B tests. We shut off experiments that were clearly losing hand-over-fist in multiple metrics*, but otherwise we ran experiments for a period of time calculated in advance (4 weeks was common).** Stopping a test early just because significance seems to have been achieved is, generally speaking, statistically invalid. Often an experiment seemed to start out strong (random chance, imagine flipping heads many times at the beginning of a fair coin sequence) but then as more data was obtained, lost statistical significance.
Note that if you run lots of A/B tests, significance will be problematic. If you run 20 independent A/B experiments at p=0.05 with one metric each, on average one experimental result will be a type-I error and potentially lead to the wrong business conclusion. It helps to have multiple confirmation of principles among many different experiments (Marcello Truzzi: "Extraordinary claims require extraordinary proof") and it's also worth educating yourself about the family-wise error rate and logical fallacies such as the appeal to novelty. There are many A/B testing best-practices, such as running continuous A/A tests. Microsoft and Google have published some useful practical information about their A/B approaches.
I'm not sure what you mean by "peak the data," but if you mean to throttle experiment data beyond an arbitrary peak (such as the examples you gave), I would recommend against that. Throttling creates sampling errors and many other bad statistical effects.
(* by doing this, we would miss any tests that got an "unlucky" start and would have eventually dug out of that deficit and become positive. You're trading experimental recall for reduced risk of losses.)
(** We used whole weeks to average out day-of-week and time-of-day effects. But this doesn't catch longer seasonality patterns. An A/B test that performs well in summer might perform poorly in winter or vice-versa.)
Until recently I worked for a high volume online travel agent with a lot of A/B testing. Tests would sometimes be stopped early, when significance was established - but only when significance was well ahead of the target and high volumes of traffic had already gone through them, so we could be very confident they would have remained significant. It's a pragmatic trade-off between the cost/lost opportunity of running tests longer than needed, vs chance of drawing wrong conclusions.
\nIn addition to the very valid points above, you also have to think about the sort of traffic the test will have seen so far and whether there is reason to think it may be atypical. For example, if your website generates a lot of visits from a once-weekly email (where customers will generally be more qualified), and that email has not yet been sent, then it would arguably be foolish to stop a test before receiving this traffic no matter how significant it may currently be.
\nIn some sectors there may also be a case to run longer to ensure you are capturing return visits. If your customers generally require several "research" visits to your website before converting, then early results may represent the conversions from customers who started this process before the trial began and not be truly indicative of likely ongoing behaviour.
\nAnother point is that the longer you run a test the more you may be able to drill down to segments of interest. OK you might get topline significance before the test completes, but running for longer may allow deeper insights, eg allowing you to drill down results on different browsers/platforms, regionally etc.
", "votes": null, "replies": [] }, { "id": 58215, "author_name": "adjgiulio", "author_url": "", "post_date": "11/17/2014 17:56:51", "content": "Thanks guys! This was really helpful!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "58097": "", "58100": "", "58101": "", "58142": "", "58215": "" }, "source": "meta" }