UX Knowledge Base

A/B Testing

Download PDF Version

A/B Testing

What is A/B Testing ?

A/B testing, also known as split testing, is a marketing technique that involves comparing two versions of the same web page or application to measure the performance of each version and see which performs better. A portion of the website visitors will be directed to the first version, and the rest to the second. A statistical analysis of the results then determines which version, A or B, performed better, according to certain predefined indicators such as conversion rate.

How is A/B testing performed on mobile apps?

A/B testing tends to be a little more complex with mobile applications. Since it is not possible to present two different versions of the application once it has been downloaded and deployed on a smartphone, the onus falls on you to come up with a workaround that allows you to modify your design, instantly update your application after the modification and then, analyze the impact of this change.

##How to create a successful A/B testing strategy?

Creating a strong A/B testing hypothesis is the first step towards a successful A/B testing program and must respect the following rules:

Hypotheses must:

  • be linked to a clearly discerned problem that has identifiable causes
  • mention a possible solution to the problem
  • indicate the expected result, which is directly related to the KPI to be evaluated

A/B test analysis

Two things should be taken care of before you analyze the test results.

  1. Sample size/Site Traffic

  2. Test duration

The statistical tests that are used to calculate the confidence level (such as the chi-square test) take into account a sample size that is close to infinity. If the sample size is low, exercise caution when analyzing the results, even if the test indicates a reliability of more than 95%. This is due to the fact that if a test with a low sample size is left active for a few more days, it will greatly modify the results. This is why it is advisable to have an optimum sample size. And even if the site traffic makes it possible to quickly obtain a sufficiently sized sample, it is recommended that you leave the test active for several days (one or two weeks) to take into account differences in behavior observed by weekday, or even by time of day.

Advanced Analytics

An A/B testing solution lets you statistically validate most hypotheses, but alone, it cannot give you a complete understanding of user behavior.

It's important to supplement A/B testing with information gathered from other means like session recordings, heatmaps, and feedback from users. This will allow you to gain a fuller understanding of your users, and crucially, help you come up with better hypotheses to test.

Heatmap and session recordings

Both of these methods provide more visibility into how various users interact with elements on a page or between pages within an application.

User Feedback

Conduct surveys to gather user-related information on their opinions, ratings and reviews, which would provide greater insight into the behaviour of different users.