How it works

An A/B test sends two versions of your campaign to a slice of your audience, waits, and then sends whichever did better to everyone else. Version A is the subject line and design you chose on the first two steps. You set up version B.

Setting one up

  1. On the last step, switch on Try two versions and send the winner.
  2. Drag the first slider to choose how much of the campaign's audience gets the test, from 10% to 50%. It starts at 30%. Half the test goes to each version, and everyone else gets the winner.
  3. Under What's different between A and B?, choose Subject lines, Design or Both. Then type the Variant B subject, or pick the Variant B design. Until version B has its own subject line or design, Send Now stays greyed out and says what's missing.
  4. Under Pick the winner by, choose Open rate, Click rate or Revenue.
  5. Drag the last slider to set how long to wait before picking the winner, from 1 to 48 hours. It starts at 4.
  6. Click Send Now. If you chose Send later, click Schedule, and the test starts at that time.
Set what's different, how to judge it and how long to wait.
Set what's different, how to judge it and how long to wait.

The summary on the last step shows the split, for example 15% to A, 15% to B and the winner to the other 70%.

Choosing how to judge it

  • Open rate suits a subject line test, but opens are a rough measure. Apple Mail counts almost every email as opened.
  • Click rate suits a design test and gives a firmer answer.
  • Revenue suits a sales email. It needs orders linked to your emails, from a connected Shopify store. Without any linked orders the two versions tie, and a tie goes to A.

What happens next

Both versions go out. The wait starts once both have finished sending, and when it ends the winner goes to everyone else. If the two do equally well, version A wins. A version that couldn't be delivered to anyone can't win. Once the test has started, the campaign can't be changed.

While the test runs, the campaign shows Testing in your list of campaigns, which opens when you click Send in the menu on the left. Open it and go to Insights to see how it's going. The A/B test card shows both versions side by side, with the one ahead marked Leading. Once the winner is picked, it's marked Winner.

Once the winner has gone out, Insights shows both versions.
Once the winner has gone out, Insights shows both versions.

If something stops the winner going to everyone else, such as sending being paused, the A/B test card says what stopped it, and it's tried again every 30 minutes.

While the test runs, the A/B test card also has Send variant A to everyone now and Send variant B to everyone now, to pick the winner yourself, and End the test, after which the people who weren't in the test don't get the email.

Good to know

  • A test needs at least three people, one for each version and someone to receive the winner. With fewer, the switch is greyed out and says why.
  • Best time can't be used with a test.
  • Test one thing at a time, so you know what made the difference.
  • On a small list, test a bigger slice, such as 50%, so each version reaches enough people to tell them apart.