Understanding email testing

A/B testing, split testing, or simply testing can be one of those rabbit holes of endless knowledge. If you commit a few hours to researching the topic online and look in the right places, your head is going to spin once you realize how deep this thing goes.

So, my intention for this module, is that you walk away with enough knowledge to have a positive impact in your business.

For that, I will…

  • Simplify the concepts
  • Give you what you need to be able to implement in your email marketing
  • Focus on what can give you positive results
  • Without trying to turn you into an A/B Testing Guru

Deal? Let’s get started with some basics before we move on to the more techie stuff…

What exactly is “A/B Testing”?

A/B Testing diagram

It’s a controlled experiment to define a hypothesis over time and learn from it.

This works the same way as a scientific experiment in which there are two groups, the controlled group that remains unaltered and the group that gets the treatment. They are kept under similar conditions for a specific period of time to prove the hypothesis.

In plain english

You test the original version against a variant to see which one gets a better response.

In this module, we’ll discuss how to plan and execute a test, what you should be testing, the mistakes to avoid and much more.

A/B vs Multivariate Testing

You’ve probably heard the term A/B Testing or “Split Testing” before, but that’s not the only type of testing.

In A/B Testing you are testing 2 versions of the similar item.

But you can also run an experiment to test multiple variables. This is called “Multivariate Testing” and the reason you don’t hear much about it is simply because most of the ESPs only offer A/B.

A good example will be testing the copy and the color of a button in a CTA, at the same time.

This is also used in web design, here is an example of a diagram designed to conduct a multivariate test:

Multivariate testing

3 things to consider before deciding to run a multivariate test:

  • It can be a little more technical than running an simple A/B
  • You need to make sure you have the necessary tools to conduct the test, like I said, most ESPs don’t offer this
  • You need to run this test with a significant amount of subscribers or you will not have enough data to come to a solid conclusion

The Goal

The goal of testing is not to prove that John is smarter that Bob.

Most people that has a little bit of knowledge in the matter would tell you that A/B Testing is done with the objective to improve results. And that will be correct. If you’re testing a subject line, the goal will be to improve your open rate.

But I will add something else: To learn.

You can move the needle with a CTA split test in your next campaign, awesome. But if you’re not learning from the results, you are wasting a huge opportunity to improve your future marketing.

How it works

As a concept, and also in the technical aspect, this stuff is pretty simple…

You have a subject line:

“The word that boosts opens… and kills clicks”

But you want to test if you can get more opens by using “You” in the copy:

“This word will boost your opens… but kill your clicks”

The original subject line is called the “control” and, as in a scientific experiment, it remains unmodified. The other version is called the “treatment,” this is the one that suffers the modification. In this case we’re adding the “you” approach to change the perspective.

There is a lot more to it, and we’ll cover that in the next lesson, but basically you run your experiment to learn what works best with your audience, before sending the campaign to the entire list.


In the next lesson, we’ll get into how to plan and execute a test.