We are seeking a 95% confidence level. What I found was that even if you are:
Correctly setting out to run a test for the recommended length of time
Avoiding peeking at the result part way through
Calling a test successful if it achieves 95% confidence
As many as one in five of your “successful” results may in fact come from having accidentally (randomly) sent more high-converting (“email”) traffic to one variant or the other. (This explains why edusson review sometimes find “successful” results when they are actually comparing two identical pages).
The glimpse of light at the end of the tunnel is that the longer you run a test for, the more the channel distribution converges (by the law of large numbers) to be the same for each variant. This means that we can fix the problem for running our tests for longer.
How long should I run my split tests for?
I set the simulation up to run tests for 2, 4 and 8 times as long as the conventional wisdom suggests. Here are the results for a range of blends of traffic where the email traffic converts between 1.2x and 1.6x as well as the rest of the traffic (in theory, I was running to a 95% confidence level):
Effect of increasing trial size on effective power
This shows that even running tests for 8 times as long as we previously thought might only get you back to a 90% confidence level (remember, the software will be reporting a 95% confidence level).
Surely there’s a better way than just blindly running our tests for 8x as long?
I haven’t been able to work out a way of just calculating how long we need to run our tests for based on the actual traffic blend and conversion rate differences by channel. Luckily though, I think I’ve stumbled across something that might give us a solution (thanks to Mat at writemypaper4me writing - whose Django A/B testing plugin we also happen to use). Mat was talking about how they routinely run A/A/B/B tests because they often find that configuration errors creep in and break their tests.
I’ve seen people recommend running A/A tests to detect setup errors or to set the sample size [PDF]. But since traffic blend can
A proposed “answer” for those who just want to run good tests
Sidenote: I don’t think I’ve quite nailed this yet – I think that we probably want a slightly different test for checking that A1=A2 and B1=B2 – I think we might want to reject more (i.e. use a lower confidence level for those tests). Can anyone think of a way of coming up with a better one?
Next time you set up an A/B test, set up two identical versions of each variant – let’s call them A1 and A2, B1 and B2. (Our hypothesis is that the conversion rate of B is better than the conversion rate of A).
Decide on the confidence level you want to see (let’s use 95% as an example). At the end of a standard-length run (note that this will be twice as long as under an A/B test since you need to send the same amount of traffic to twice as many pages), using standard measurements of confidence:
- If we see a significant difference in the conversion rate of A1 and A2 (at the 95% level) we call the test a dud
- Likewise if we see a significant difference in the conversion rate of B1 and B2
If both of those pass, we declare B a winner if we also see a significant difference between the conversion rate of A1 and B1 (at the 95% level).
This methodology removes the need to run exceptionally long tests to be confident in rarely seeing skewed results. Instead, it focuses on discarding tests that appear to have been skewed by uneven traffic mixes leaving only real results we can confidently put live.
Will this work?
It’s hard to calculate the exact impacts of doing things this way, so I ran a follow-up simulation using the same approach described above using my proposed A/A/B/B methodology (you can find the code here). In the chart below (built on the same assumptions described above – but running A/A/B/B tests), you see:
- Increasing numbers of successful tests (blue bar) – the longer we run the trial for, the more often we gain confidence in our outcome
- There are a bunch of tests we have to discard because either the As or the Bs haven’t converged (green bar)
Useful information: