The latest version of the Gozaimasu module of our premium WordPress auto-responder plugin supports A/B testing of message subjects. (This is different than the subscribe form design A/B tests. Check them too, they are great. It’s also different than split testing whole autoresponder messages.)

What Does It Do

A/B testing is a very powerful method to figure out what subjects in your messages draw more attention, email reads and clicks. Sending the same email but with different subjects to different subscribers can help you figure out what is the best subject.

Normally you would run A/B test for some time, then add or remove different subjects and at the end you will stop the test and use only the subject that works best.

How Exactly Does It Work

The A/B subjects feature picks one random subject for each email the software sends from the subjects which you have defined for the test. This is completely random. To avoid unnecessary queries and server overload it will not guarantee that each subject is picked equal number of times.

Here’s a basic example. You can create a welcome message in an autoresponder campaign. You want a catchy subject but you are not sure whether this is a good idea. So you can decide to run an A/B test between these two (or more) subjects:

Welcome to Our Newsletter!

or

Want To Learn About The Best Fitness Practices? You Came To The Right Place!

Some of your subscribers will receive the first subject, some the second. By looking at the reports: read stats, unsubscribe stats (and if you have the Intelligence module you can add trackable links for click stats), you will see which of these subjects performs better. This is simple and efficient.

How To Set It Up

The feature is available both for autoresponder messages and newsletters so let’s just see it on newsletters. On the Add/Edit Newsletter page you will see a checkbox for A/B Test next to the newsletter subject.

Clicking on it shows a new field and transfers your current subject to it. Clicking on the + sign next to it lets you add more variants, as many as you want:

Save the newsletter and now when it’s sent, it will pick these subjects randomly instead of the default Newsletter Subject field (empty fields will be ignored).

Absolutely the same works for autoresponder email messages.

Then in the Newsletter or Email message Reports page you will see a new link “Reports per A/B Subject” and will be able to see how many emails are sent with each subject, how many are read, the % read rate, and for Intelligence module owners also the trackable link clicks:

It’s worth noting that “read” stats rely on a tracking image. Since some email programs disable messages by default, the read number is not 100% reliable and sometimes shows slightly less than the actual number of read emails. It’s still a great way to identify which subject performs better.

Trackable link clicks will count one click per email message regardless how many links you have placed inside and how many times each reader clicked them. You want to know which subject makes people actually read the message and now if they clicked 10 links inside – counting all the clicks would defeat the purpose of the report.

In your Advanced Stats page, the Unsubscribe stats tab will also show reports based on A/B subjects for these email messages and newsletters which currently have an A/B test enabled:

You may wonder why the number of emails sent total for this newsletter is higher than the sum of emails per A/B subject. This is because we were sending this newsletter for a while without running A/B tests and started using tests later.

The unsubscrbe stats per A/B subject are very useful: they can show you what subjects irritate the subscribers – probably because of looking spammy or maybe over promising (and then under delivering in the content of the email).