4 Steps for Successful A/B Testing

A/B testing is a term that describes simple randomized experiments with two variants. It is not a new technique (actually, it has been available for decades); however, some A/B experiments are not built correctly, so the decisions made that are based on them do not generate the expected results.

To avoid making marketing and business decisions based on faulty A/B test results, I’ve created four steps to make sure you’re A/B tests are successful:

1. Define the Test Goal

   You need to know when to stop the test and evaluate if it was successful. If you are testing a landing page, the objective should be to increase the number of products purchased through it, if you are evaluating a sign-up form, the objective is to increase the number of registered users, etc. This is why it’s important to know the AS-IS, the TO-BE and translate those to a conversation rate.

2. Define the Conversion Rate

   Based on the Test Goal, you need to define the conversion rate that is appropriate for it. For the landing page scenario, the conversion rate could be the number of “Add to Cart” / number of visits or it could go deeper and be number of purchases/visits (through the landing page). It is useful to have the statistics of the original landing page (usually called ‘control’) so you can determine if the % increase is relevant.

3. Define the Page Variations

   Based on the original page, you have to define the sections to change and its variations or if you want to change the complete page look and feel. There are two different types of test: Multivariate test and Page Level test.

   1. Multivariate Test changes a specific section(s) of the page. If we are testing multiple variations we need to define all the combinations to test (e.g. Section 1/ Variation 1 with Section 2/Variation 1, etc.). These are some sections candidates to include in the multivariate test:

      • Headings (size, color wording)

      • Images (size, placement, different images)

      • Content (amount, wording, font, size, placement of content on the page)

      • Call to Action buttons such as: buy now, sign-up, subscribe buttons. Try these with different sizes, colors, in different places on the page and with different wording.

   2. Page Level Test, this test is use to compare the effectiveness of two different versions of a landing page. Instead of replacing a single section, all of the page (or most of it) is changed to analyze the user acceptance against the page variation. This type of testing is most useful when you’re planning a website redesign.

4. Define the percentages to choose a winner (variation succeed / total samples %)

   It is important to define the percentage of success to select a variation as a winner (e.g. > 65% of the total samples) and how long that percentage should be maintained to be consider as a final result (e.g. > 3 days). You should keep the tests running for an appropriate amount of time to ensure its reliability (statistical confidence).

BONUS POINTS

• Do not test page variation “A” first and then page variation “B”. All the variations should to be tested simultaneously.

• Do not include regular customers in your test samples. Consider using only new customers/users in your tests.

• Don’t switch variations on your returning users. Make sure to show them the same variation of testing when they come back to your site while you’re testing.

• Make your variations consistent across the complete site. If you change the “Add to Cart” button, this change has to be reflected everywhere.

A/B Testing is vital to making sure your marketing efforts are effective and that your users are getting the best possible experience on your website. To get the most out of A/B testing, make sure you follow these steps and always keep track of your data!

Reference: www.territoriocreativo. es/etc/ 2009/02/ experimentando-con-el-test-ab. html

Image courtesy of TheBusyBrain via Flickr Creative Commons