How to create an A/B test

How to create an A/B test

In Mailrelay you can create A/B tests for newsletters, for a small subset of subscribers, and analyze the results obtained with this group, and then send the winning campaign to all the remaining contacts.

This will help us to get the best results from the campaigns, without having to do practically anything.

TABLE OF CONTENTS

• Step 1: Decide on the type of A/B test
• Step 2: Preparing the A/B test
• Step 3: A/B test forthe sender
  • Step 3.1: Decide manually
  • Step 3.2: Pick the newsletter with most clicks
  • Step 3.3: Pick the newsletter with most impressions
• Step 4: A/B tests for subject lines
  • Step 4.1: I will decide manually
  • Step 4.2: Pick the newsletter with most clicks
  • Step 4.3: Pick the newsletter with most impressions

Step 1: Decide on the type of A/B test

A/B tests can be created based on two conditions:

• Sender: This option allows you to use two senders to find out which one offers the best click and open rates.
• Subject line: In this case, two subjects are used to find out which one offers a better click or open rate.

Step 2: Preparing the A/B test

To start an A/B test, it is essential that you have already created a campaign that will be used in the A/B test.

In this case, simply go to Campaigns >>A/B test.

After that, click on the Add button to create the new test. Preparing the A/B test for your newsletter Next, you will need to choose the newsletter you want to use for the A/B test. This can be done in the Base Campaign option. Once you have selected it, just click Next to continue with the process.

After that, you have to select the groups or segments to which you want to send the newsletter, for this in the option To whom you want to send the test, you must select one of the two possible options: Groups or Segment. After that, just below, you can include the groups or segments (a single group or segment is also allowed).

Please note that in order for the system to allow you to create the A/B test, the group or segment must have at least 10 subscribers, or you will not be able to send the A/B test.

When everything is ready, click Next and you will be taken to the final A/B test setup screen, where you will need to decide between several options according to your preference:

• What would you like to test: where you should click on Sender or Subject, depending on the type of A/B test you want to create.
• What percentage of recipients will receive the tests? Here you should indicate the percentage of the subscribers you will use for the test (the exact number is calculated by the system based on the total number of subscribers in each group/segment).
• How can I decide which campaign was the winner? This option allows you to automate (if you wish) the way the system will act when the A/B test is completed.

At this point, you will be able to choose between several options:

1. I will decide manually

2. Select the newsletter with the highest number of clicks.

3. Select the newsletter with the highest number of impressions.

Step 3: A/B test forthe sender

For a better understanding, we will provide the exact steps for creating an A/B test with different senders and how to make use of the different options when deciding which was the winning newsletter (the one that will be sent to the rest of the contacts who are waiting to receive the newsletter).

Step 3.1: Decide manually

Let's get started, select Sender and you will have the option to select the number of senders that will be used in the A/B test, you can choose either between 2 or 3 senders for the test.

After that, you have to decide how you want the winning campaign to be decided; in this first case, select I will decide manually and click Next.

This will take you to the window where you have to select the senders you want to use for the test; in case you opted for two, you will have to fill in the fields Sender A and Sender B; if you have selected three, you must also include a third in the Sender C field.

To continue, click Next and you will see a screen with a summary of the A/B test you have created.

If everything is correct, click Send it now to proceed with the A/B test with these values or click the Back button to return to the initial setup screen.

After sending it, you can go to Campaigns >>A/B Test and view the A/B test statistics by clicking on the blue magnifying glass next to the A/B test.

To finish, just click on the blue magnifying glass and you will see the window where you will have to decide which of the A/B tests is the winner based on the impressions and clicks generated by each of the campaigns that participated in the test. When you are sure, click on the green button (Use this combination) and the test will be completed with the selected newsletter.

You can always cancel the test by clicking Cancel

Step 3.2: Pick the newsletter with most clicks

To do this, select Sender and you will be given the option to select the number of senders that will be used in the A/B test, allowing you to choose between 2 or 3 senders for the test.

After that, you have to decide how you want the winning campaign to be decided; in this case, select the newsletter with the highest number of clicks.

When you select this option, a drop-down menu will be displayed in which you must select the number of days to wait for the A/B test to be completed. By default, the system is set to 2 days, but you can freely change between hours and days. With everything set to your liking, click Next.

This will take you to the window where you will have to select the senders that will be used in the test.

If you have selected two senders, you must fill in the fields Sender A and Sender B. If you have selected three senders, in addition to them, you must include one more sender in the Sender C field.

To continue, click Next and you will see a screen with a summary of the A/B test you have created.

If everything is correct, click on Send it now to proceed with the A/B test with these values or click on the Back button to return to the settings screen.

After it was sent, you can go to Campaigns >>A/B Tests and view the A/B test statistics by clicking on the blue magnifying glass next to each A/B test combination.

You can always cancel the test by clicking Cancel. You can always choose to send it manually, for this you will have to click on Cancel the automatic decision. I will decide manually.

Step 3.3: Pick the newsletter with most impressions

To do this, select Sender and you will have the option to select the number of senders that will be used for the test. You can choose between 2 or 3 senders.

After that, you must decide how you want the winning subject to be decided; Pick the newsletter with most impressions.

When you select this option, a drop-down menu will be displayed in which you must select the number of days to wait for the A/B test to be completed.

By default, the system offers 2 days, but you can freely change between hours and days. With everything set to your liking, click Next.

This will take you to the window where you will have to select the senders you want to do the test with; if you have chosen two, you will have to fill in the fields Sender A and Sender B.

If you have selected three senders, in addition to them, you will have to include one more sender in the Sender C field.

To continue, click Next and you will see a screen with a summary of the A/B test you have created.

If everything is correct, click on Send it now to proceed with the A/B test with these values or click on Back to return to the settings screen.

Once the test is sent, you can go to Campaigns>>A/B Tests and view the A/B test statistics by clicking on the blue magnifying glass next to each A/B test combination.

You can always cancel the test by clicking Cancel, as well as send it manually by clicking Cancel automatic decision) I will decide manually.

Step 4: A/B tests for subject lines

For a better understanding, we will look at the precise steps for creating an A/B test based on the subject line and how to use the different options when deciding which is the winning newsletter (the one that will be sent to the rest of the contacts who are waiting to receive the message).

Let's get started. Pick Subject and you will have the option to select the number of subject lines that will be used in the A/B test. You will be able to choose between 2 or 3 test subjects.

Step 4.1: I will decide manually

After that, you have to decide how you want the winning subject line to be decided; in this first case, select I will decide manually and click Next.

This will take you to the window where you will have to include the subject lines you want to test; if you have chosen two, you will have to fill in the Subject A and Subject B fields, if you have chosen three, you must include one more subject line in the Subject C field.

To continue, click Next and you will see a screen with a summary of the A/B test you have created.

If everything is correct, click Send it now to proceed with the A/B test with these values or click the Back button to return to the settings screen. After sending it, you can go to Campaigns >>A/B Test and view the A/B test statistics by clicking on the blue magnifying glass next to the A/B test.

To finish, just click on the blue magnifying glass and you will see the window where you will have to decide which of the two A/B tests is the winner based on impressions and clicks generated by the campaigns used in the test. When you are sure, click on the green button (Use this combination) and the test will be completed with the selected newsletter.

You can always cancel the test by clicking Cancel.

Step 4.2: Pick the newsletter with most clicks

If you decide to use the Pick the newsletter with most clicks option, the A/B testing process will be very similar.

When you select this option, you will be presented with a drop-down menu in which you must select the number of days the system should wait for the A/B test to be completed. By default, the software offers 2 days, but you can freely change it between hours and days. Once you have selected the time range, click Next.

This will take you to the window where you will have to include the subject lines you want to test; if you have chosen two, you will have to fill in the Subject A and Subject B fields, if you have chosen three, you must include one more subject line in the Subject C field.

To continue, click Next and you will see a screen with a summary of the A/B test you have created.

If everything is correct, click Send it now to proceed with the A/B test with these values or click the Back button to return to the settings screen. After sending it, you can go to Campaigns >>A/B Test and view the A/B test statistics by clicking on the blue magnifying glass next to the A/B test.

In the test window, you will see a blue box indicating the time and day when the A/B test will be completed (this depends on the time range you selected earlier), as well as the details of the A/B test.

You can always cancel the test by clicking Cancel, as well as send it manually by clicking Cancel automatic decision. I will decide manually.

Step 4.3: Pick the newsletter with most impressions

If you decide to use the Pick the newsletter with most impressions option, the A/B testing process will be very similar to the previous two cases.

When you select this option, a drop-down menu will be displayed in which you will have to indicate the number of days you should wait for the A/B test to be completed. By default, the system offers 2 days, but you can freely change it between hours and days. Once you have selected the time range, click Next.

This will take you to the window where you will have to include the subject lines you want to test; if you have chosen two, you will have to fill in the Subject A and Subject B fields, if you have chosen three, you must include one more subject line in the Subject C field.

To continue, click Next and you will see a screen with a summary of the A/B test you have created.

If everything is correct, click Send it now to proceed with the A/B test with these values or click the Back button to return to the settings screen. After sending it, you can go to Campaigns >>A/B Test and view the A/B test statistics by clicking on the blue magnifying glass next to the A/B test.

In the test window, you will see a blue box indicating the time and day when the A/B test will be completed (this depends on the time range you selected earlier), as well as the details of the A/B test.

You can always cancel the test by clicking Cancel, as well as send it manually by clicking Cancel automatic decision. I will decide manually.