A/B Testing

Digital Marketing - Study Notes:

What is A/B testing?

A/B testing, also known as split testing, it's a method of comparing two versions of an email against each other to determine which one performs better. A/B testing is an experiment where two or more variants of an email are shown to users at random. An analysis is then used to determine which variation performed better for a given goal, such as opens or clicks.

So, you can have multiple variations, but you test the same element. If you have five versions of an email, everything remains the same except the one element that you are trying to improve upon.

Take a look at this graphic here to get an idea of how it works. There are two variations, and the metric being focused on is the click.

So, again, A/B testing can be used to test almost any element of an email, but only perform one test at a time to get accurate results. In this scenario, we are testing two emails which are the exact same, but have two different calls to actions to see which one gets more clicks. We send 20% of our overall audience to each email, wait one hour to view the results, and then whichever one has gotten the more clicks, the rest of the subscriber list will receive it.

The test group that got the underperforming email is an unfortunate side effect of your A/B test. However, in your next send with the improved version, you'll know that it will go to everyone, and perform better. This is also why it's extremely important to have a continuously growing database. Without a healthy size database, you will never have a large enough list to actually be able to do your testing.

Best practices

A/B testing best practices are very important to know:

• Only test one variable at a time.
• Split your groups equally and randomly.
• Test early and test often for the best results.
• Test as large a sample as you can for more accurate results.
• Trust the data collected, not your gut instinct.

If you're wondering about sample size, I would advise you to go to the blog of ConversionXL (cxl.com/blog) to understand in more depth what sample audience size can and should look like, where it's too small, and all those other questions you may have.

Advantages

A/B testing is one of the most common testing methods in email marketing, as it is, at the end of the day, quite quick and easy to implement, and there's so many advantages.

• You better understand each of your segments. You can begin to see why this group is different from that group, and why they should be treated as such.
• You determine the best subject lines so you get to know the language that works for them. This will impact all of your marketing, not just your email.
• You have improved content. You will start to filter out the fluff that does not matter to your subscribers, and focus on what really delivers value to them.
• It increases conversion rates because you're sending the right thing, genuinely, at the right time.
• This all results in more sales. Who can say no to that?