A/B Testing - Quick Guide

A/B Testing – Overview

A/B Testing (also known as Split testing) defines a way to compare two versions of an application or a web page that enables you to determine which one performs better. It is one of the easiest ways to analyze an application or a web page to create a new version. Thereafter, both these versions can be compared to find the conversion rate, which further helps in finding the better performer of these two.

Example

Let us assume that there is a web page and all the traffic is directed to this page. Now as a part of A/B Testing, you have made some minor changes like headlines, numbering, etc. on the same page and half of its traffic is directed to the modified version of this web page. Now you have version A and version B of the same web page and you can monitor the visitor’s actions using statistics and analysis to determine the version that yields a higher conversion rate.
A conversion rate is defined as the instance, when any visitor on your website performs a desired action. A/B Testing enables you to determine the best online marketing strategy for your business. Take a look at the following illustration. It shows that version A yields a conversion rate of 15% and version B yields a conversion rate of 22%.

A/B Testing – How it Works

You can monitor the visitor’s actions using statistics and analysis to determine the version that yields a higher conversion rate. A/B Testing results are usually given in fancy mathematical and statistical terms, but the meaning behind the numbers are actually quite simple. There are two significant methods through which you can check conversion rates using A/B Testing −

• Sampling of Data
• Confidence Intervals

Let us discuss these two methods in detail.

Sampling of Data

The number of samples depend on the number of tests performed. The count of conversion rate is called a sample and the process of collecting these samples is called as sampling.

Example

Let us say you have two products A and B, you want to collect sample data as per its demand in the market. You can ask a few people to choose from product A and B and then request them to participate in a survey. As the number of participant’s increase, it will start showing a realistic conversion rate.
There are various tools that can be used to determine the correct number of sample size. One such free tool available is −
http://www.evanmiller.org

Confidence Intervals in A/B Testing

Confidence interval is the measurement of deviation from the average on the multiple number of samples. Let us assume that 22% of people prefer product A in the above example, with ±2% of confidence interval. This interval indicates the upper and lower limit of the people, who opt for Product A and is also called as margin of error. For best results in this average survey, the margin of error should be as small as possible.

Example

Let us assume that in Product B, we have added a minor change and then performed A/B Testing on these two products. Confidence interval product A and B are 10% with ±1% and 20% with ±2% respectively. So this shows that a minor change has increased the conversion rate. If we ignore the margin of error, conversion rate for test variation A is 10% and conversion rate for test variation B is 20%, i.e. a 10% increase in the test variation.
Now, if we divide the difference by control variation rate 10% ÷ 10% = 1.0 = 100%, it shows an improvement of 100%. Hence, we can say that A/B Testing is a technique based on mathematical methods and analysis. There are various online tools that can be used to calculate A/B significance.
http://getdatadriven.com

A/B Testing – Why to Use?

A/B Testing is used to make business decisions based on the results derived from data, instead of just making predictions. It allows you to create variations of your website or app and then helps you to confirm or discard your decision to make changes.
This testing allows you to optimize your site or app in such a way that it increases the conversion rates. A higher conversion rate means getting more value from your existing users instead of having to pay more on traffic acquisition. A/B Testing can also help you to change other factors in business like changing work culture, etc. It helps you to use mathematical data and statistics to determine the direction of your product variations.
Either you are a designer, a business analyst, or a developer, A/B Testing provides a simple way to use the power of data & statistics to reduce risks, improve results, and become more data-driven in your work.
To run an A/B Test, you should consider the following points −

• Always perform A/B Testing if there is probability to beat the original variation by >5%.
• Test should be run for a considerable amount of time, so that you should have enough sample data to perform statistics and analysis.

A/B Testing also enables you to gain maximum from your existing traffic on a webpage. The cost of increasing your conversion is minimal as compared to the cost of setting up the traffic on your website. The ROI (return on investment) on A/B Testing is huge, as a few minor changes on a website can result in a significant increase of the conversion rate.