UX Analytics Guide

UX Testing

Download PDF Version

Fundamentals of UX Testing


Building a mobile app is only half the battle. Once that's been accomplished it's time to move on to optimising your app and monitoring the behaviour of your app users. And with over 2.7 billion smartphone users and over 4 million apps worldwide, the competition is bound to be fierce. In order to stay relevant, businesses have to remain sharp and offer a good mobile app experience to their customers. But how can they know if they're on the right track?

Here we are to help you optimise your app user experience by helping you with some of the pressing issues like -

  1. Who are your users?

  2. What is their onboarding experience?

  3. What are they looking for?

  4. How is the new upgrade received?

  5. Why are they not converting?

  6. What can we do to enhance the customer experience?

And more...

Difference between UX Testing and Usability Testing

User experience testing mainly consists of collecting qualitative and quantitative data from the user, whilst the user is subject to all or any aspects of a service or product. Qualitative or quantitative data is collected from the user to improve ux and is done using various methods and usually with the aid of a tool or service.

In simpler terms, user experience is how you feel about every interaction you have with what's in front of you in the moment that you're using the app. It includes measuring users' emotions, gaze movements, preferences, and all key details of behavior during and after use of the product.

Usability is the way of how a product may be used by users to succeed in specified goals. It addresses whether or not you're able to achieve a task or goals with a product or service. Usability testing is aimed to uncover what portion the merchandise (app, website) is easy to use, understandable, is it being able to satisfy the users needs effectively.

Many people confuse UX design with usability and the other way around. However, mobile app usability is a facet of UX that plays into the general relationship between user and the product. UX encompasses all aspects of a user's perception of a mobile app, including usability.

Importance of UX Testing

Data only shows part of the story. If you're only going by what analytics have to show you, then you're essentially guessing. Sure, it can be an educated, highly informed guess -- but you won't know exactly why things are failing on your site until you see real people using it.

A poor mobile app experience led to the below mentioned user reactions:

  • 48% are less likely to use the mobile app

  • 34% switch to a competitor's mobile app

  • 31% tell others about the poor experience

  • 31% less likely to purchase from your company

  • 24% have a negative overall perception of the company

UX testing gives companies unparalleled insights into the otherwise hidden lives of app users. UX analytics usually comes within the sort of a software that integrates into companies' existing mobile apps to capture, store, and analyse the information. This data is vitally important to marketing, sales, and product management teams who use it to form more informed decisions. Without a UX analytics solution, companies are left flying blind. They're unable to tell what users engage with and why they leave. Thereby, making your app more usable.

So analyzing the product's user-flow and overall UX will allow designers to discover many pain points and frustrations---to walk a mile in the users' shoes and uncover opportunities that will improve the product's user experience overall.

UX testing advantages -

  1. Helps uncover UX issues from an early stage

  2. Helps improve end-user satisfaction

  3. Makes your app highly effective and efficient

  4. Helps gather true feedback from your target audience who actually use your app. You do not need to rely on "opinions" from random people

  5. Improves retention and thereby your revenue

UX Testing Checklist

The primary goal of the UX testing is to seek out crucial UX problems before the app is launched. Following things need to be considered to make a testing success:

  • Start early - Start the UX testing during the early stage of design and development

  • Target audience -Select the appropriate users to test the app(Can be experts/non-experts users/50-50 of Experts and Non-Experts users)

  • Testers need to concentrate on critical & frequently used functionalities of the system.

  • Educate designers and developers that these testing outcomes are not a sign of failure but a sign of improvement

UX metrics you should be tracking

User experience is very subjective by nature. We can measure its effects on product engagement (e.g. page views, time spent, etc.) and conversions (signups). But how can we directly measure and benchmark UX?

The old paradigm of mobile analytics is geared more towards measuring progress against business goals. While still useful, they're lagging indicators of UX decisions.

Common metrics include:

  • Page Views- Number of pages viewed by a single user.
  • Uptime- The percentage of time the website or application is accessible to users.
  • Latency- The amount of time it takes data to travel from one location to another.
  • Active users- The amount of unique users to interact with the app like daily active users(DAU), weekly active users(WAU), monthly active users(MAU) etc.
  • Earnings- Revenue generated by the app.

While businesses should still track these metrics, it has to be remembered that they lack context for measuring UX. For example, an average time on an app of 5 minutes might mean users are highly engaged -- or just aren't finding the content they need.

User-centered metrics are hard to come by. Either the parameters are too generic to be useful, or too specific to be applicable across the board.

A/B Testing

What is A/B Testing ?

A/B testing, also known as split testing, is a marketing technique that involves comparing two versions of the same web page or application to measure the performance of each version and see which performs better. A portion of the website visitors will be directed to the first version, and the rest to the second. A statistical analysis of the results then determines which version, A or B, performed better, according to certain predefined indicators such as conversion rate.

How is A/B testing performed on mobile apps?

A/B testing tends to be a little more complex with mobile applications. Since it is not possible to present two different versions of the application once it has been downloaded and deployed on a smartphone, the onus falls on you to come up with a workaround that allows you to modify your design, instantly update your application after the modification and then, analyze the impact of this change.

##How to create a successful A/B testing strategy?

Creating a strong A/B testing hypothesis is the first step towards a successful A/B testing program and must respect the following rules:

Hypotheses must:

  • be linked to a clearly discerned problem that has identifiable causes
  • mention a possible solution to the problem
  • indicate the expected result, which is directly related to the KPI to be evaluated

A/B test analysis

Two things should be taken care of before you analyze the test results.

  1. Sample size/Site Traffic

  2. Test duration

The statistical tests that are used to calculate the confidence level (such as the chi-square test) take into account a sample size that is close to infinity. If the sample size is low, exercise caution when analyzing the results, even if the test indicates a reliability of more than 95%. This is due to the fact that if a test with a low sample size is left active for a few more days, it will greatly modify the results. This is why it is advisable to have an optimum sample size. And even if the site traffic makes it possible to quickly obtain a sufficiently sized sample, it is recommended that you leave the test active for several days (one or two weeks) to take into account differences in behavior observed by weekday, or even by time of day.

Advanced Analytics

An A/B testing solution lets you statistically validate most hypotheses, but alone, it cannot give you a complete understanding of user behavior.

It's important to supplement A/B testing with information gathered from other means like session recordings, heatmaps, and feedback from users. This will allow you to gain a fuller understanding of your users, and crucially, help you come up with better hypotheses to test.

Heatmap and session recordings

Both of these methods provide more visibility into how various users interact with elements on a page or between pages within an application.

User Feedback

Conduct surveys to gather user-related information on their opinions, ratings and reviews, which would provide greater insight into the behaviour of different users.