UX Knowledge Base

A/B Testing

Download PDF Version

A/B Testing

What is A/B Testing ?

Also known as split testing, is a marketing technique that involves comparing two versions of a web page or application to see which performs better. These variations, known as A and B, are presented randomly to users. A portion of them will be directed to the first version, and the rest to the second. A statistical analysis of the results then determines which version, A or B, performed better, according to certain predefined indicators such as conversion rate.

How do we perform A/B testing on mobile apps?

A/B testing is more complex with mobile applications. This is because it is not possible to present two different versions once the application has been downloaded and deployed on a smartphone. Workarounds exist so that you can instantly update your application. You can easily modify your design and directly analyze the impact of this change.

Here is how you can do Create a strong hypotheses- A correctly formulated hypothesis is the first step towards a successful A/B testing program and must respect the following rules:

Hypotheses must:

  • be linked to a clearly discerned problem that has identifiable causes

  • mention a possible solution to the problem

  • indicate the expected result, which is directly linked to the KPI to be measured

A/B test analysis

Two things should be taken care before you analyze the test results. 1. Sample size/Site Traffic

  1. Test duration

The statistical tests used to calculate the confidence level (such as the chi-square test) are based on a sample size close to infinity. Should the sample size be low, exercise caution when analyzing the results, even if the test indicates a reliability of more than 95%. With a low sample size, it is possible that leaving the test active for a few more days will greatly modify the results. This is why it is advisable to have a sufficiently sized sample.

And even if the site traffic makes it possible to quickly obtain a sufficiently sized sample, it is recommended that you leave the test active for several days to take into account differences in behavior observed by weekday, or even by time of day. A minimum duration of one week is preferable, ideally two weeks. In some cases, this period can even be longer

Advanced Analytics

An A/B testing solution lets you statistically validate certain hypotheses, but alone, it cannot give you a sophisticated understanding of user behavior.

It's therefore essential to enrich A/B testing with information provided by other means. This will allow you to gain a fuller understanding of your users, and crucially, help you come up with hypotheses to test.

Heatmap and session recordings

Both of these methods offer more visibility on how users interact with elements on a page or between pages.

User Feedback

User feedback through surveys, opinions, ratings and reviews could help you