5 Questions to Ask Before A/B Testing

When I first learned about what A/B testing (or split-testing) was, I thought it seemed like a pretty straight-forward concept. Just send half of your visitors to one page and half to the other and then see which one converts better– how hard could it be? However as soon as I started actually implementing split tests I realized how complicated the process really is.

Between the planning of the strategy, the technical set-up, and working out all the statistics to make sure you’re getting the right data, there’s a ton to think about! Below are 5 of the most important questions to ask to make sure you cover all your bases when optimizing.

What are your goals?

First you need to know what your goals are with your test, and what measurable impact you are trying to make. Is it to get more sales, more leads, or just get your visitors to click on the call to action button?

Note that the amount of traffic you plan on getting may impact the goal that you choose. If you think you’re only going to get a handful of sales, then you better pick a different goal so that you actually have enough data to optimize (more on this later).

What will you test?

How different are the variations going to be? Will they be two completely different pages or are you just testing something small like a button colour or headline copy change?

A smaller change will likely have less of an impact, but can be good if you already know that the page converts well. It can be difficult to look at a page and know what areas to test, so your analytics will help a lot here. This is where experience in conversion optimization or user experience design also comes in handy because you know what has worked in the past.

How much data do you need?

Before running your test, you should have an idea of how much traffic and how many conversions you will be able to get. This will help you decide how long you want to run your test for. The more data you have, the more certain you can be of the results.

If you don’t have a large audience already, you will likely have to consider running paid traffic to the split test, and this will require creating ads and determining a budget. If your traffic is expensive, you will have to decide whether you want to spend more and get more learnings, or spend less and be less certain of your results.

You also need to decide how you want to split your traffic. It could be 50/50 or 70/30 or maybe you have more than 2 variations are splitting it 80/10/10. However you split it, you still need to ensure that each variation gets enough data for you to reach significance. 

Check out Evan Miller’s calculator for finding out how much traffic you need for your test.

What tools will you use to run your test?

You will need a way to actually run your split-test and to track the results. There are tools out there like Unbounce or Click Funnels that allow you to actually build the page variations and split your traffic. There are also more advanced tools like Optimizely and Visual Website Optimizer that give you more insight into the data behind the test. My personal favourite is Google Optimize, and even better, it’s free! 

What outside factors should you be aware of?

No matter how careful you are with planning out your test, there will always be factors that affect your test that are outside of your control. This includes the quality of your traffic — for example email subscribers will be more likely to convert than cold traffic from advertising. Also the time of day, week, or year will have an effect. You may have a page that converts very well right before Christmas but doesn’t convert for you the rest of the year.

Audience demographics, page loading times, and the device the user is on are just a few more of the millions of factors that can affect your test. Be aware of what factors may have had an impact on your results and keep them in mind when testing. However, there is no way to eliminate these completely; just try to be aware of what they are and how they might have impacted your results.

And lastly, always keep testing and optimizing! Your results are going to change over time and what works today might not work next year or even next month. If you’re doing conversion optimization, you should try to have tests running at all times to take advantage of your website traffic.

Your success is not likely going to come from one massive win, it’s going to be layering a lot of small wins on top of each other to make that high-converting landing page!

Happy testing!


Share this post: