This article explains how to conduct a subject line split test.

Subject line split testing is a way of working out which of two or three subject lines is the most effective at encouraging people to open your mailing. It's also a great way to learn what works and what doesn't, so that you can continuously move toward improving your conversion rates. You can also use a content split test to test entire campaigns.

	In This Article
	Who Can Create a Subject Line Split Test? Video Tutorial Create a Subject Line Split Test Manually Selecting a Winner Frequently Asked Questions Additional Resources

Who Can Create a Subject Line Split Test?

Depending on your account type, certain users may not be able to create split tests. Users in the Editor, Author, and Response Only roles cannot create content split tests. Users in the Parent, Manager, Administrator, and Full roles can create content split tests.

For legacy* campaigns, users who cannot send campaigns (Editor and Author) also cannot create subject line split tests. For campaigns created in the new drag and drop editor, Editor and Author users can create subject line split tests, but not content split tests.

Watch a Quick Video Tutorial on Split Testing

Creating a Subject Line Split Test

In order to conduct a subject line split test, you'll need to create a campaign. Once you've done that and you're ready to send, follow the instructions below.

☑	Tip
	Legacy* subject line split tests cannot be sent to subscriptions, only to groups and segments. If you would like to send a split test to a subscription, you will need to create a campaign in the new drag and drop editor instead.

Drag and Drop Editor

1. Click on the Review button.
2. Click on the Subject dropdown arrow.
3. Click on the Switch to split test button.
4. Enter two or three subject line variations.
5. (Optional) Check or uncheck the "Autosend the winning variation" box to automatically send the most popular subject line when the test period ends. We recommend leaving it checked.
6. (Optional) Click on the Advanced dropdown arrow and then click on the Add Google Analytics button to add Google Analytics to the campaign.
7. Fill out all the required information in the "Send time", "Audience", and "From" sections.
8. Click on the Review & send button.
9. Confirm the information in the pop-up window.
10. Click on the Send campaign button or the Schedule campaign button.

Legacy* Editor

1. Click on the Review & send button.

2. On the next screen, click on the Switch to split test button.

   

3. Enter two or three subject line variations.

   

4. (Optional) Check or uncheck the "Autosend the winning variation" box to automatically send the most popular subject line when the test period ends. We recommend leaving it checked.
5. (Optional) Connect Google Analytics to the campaign by clicking on the Add Google Analytics button on the left side and following the prompts.
6. Select your audience for the campaign and double check your sender name, sender email, and reply-to email.
7. Review all of your choices and click on the Start split test now button to begin testing immediately or click on Schedule split test to start later and follow the prompts.

Split Test Process

When the test starts, the mailing will be sent to 20% of the recipients selected. For example, if the total number of recipients is 5,000 contacts, the randomly selected test group will be 1,000 contacts. However, in order for the split test to function, you must send it to at least 30 recipients. Subject line split tests run for four hours, during which time opens are tracked to determine the winning subject line.

There is also a small box that is checked by default, this is set to automatically send the winning variation to the remaining 80% of your recipients when the test completes. You can uncheck this option, but we recommend leaving it checked because if you forget to manually choose a winner after the test has ended, the campaign will not be sent to the remaining recipients that you have selected.

✎

Note

When Emma automatically sends the winning campaign, it will identify the winner based on the response data from the four hour test period. It is possible that the response data will change after more time elapses so that it looks like the "losing" campaign was sent instead of the winning campaign. When this occurs, it is because the "losing" campaign had more engagement during the four hours test period and so was actually the winning campaign at the end of the test period.

Manually Selecting a Winner

While the test is running, you can check to see which subject line is winning, and optionally select the winner yourself. To check on progress while a test is running, open the Response section of your account. Your Mailings page will open on the In Progress tab. Click on the name of the mailing to view the test results page.

From here you have the option to end the test early, for example, if there's a clear front runner, or if you need to get the email sent earlier than planned. Determine the winning option at any time during the test, simply click End test & send this variation next to the subject line you want to send to the remaining recipients.

Frequently Asked Questions About Subject Line Split Tests

Can I send a split test to a subscription?

It depends. If the campaign was created in the drag and drop editor, then yes, you can choose a subscription as the audience of a subject line split test. However, if the campaign was created in the legacy* editor, then no, you cannot choose a subscription as the audience of a subject line split test.

 

Why does my winning subject line have fewer opens than the other subject lines?

After the winning version has been sent, opens continue to be recorded for the 20% of recipients in the test group. Sometimes this can result in another variation of the subject line outperforming the version that was in the lead when the test period ended.

 

How is the winner chosen if my subject line split test is 50/50?

In the case of a true tie, the winner is chosen based on the highest unique open rate for the variation at the end of the testing phase. However, response data continues to track for losing variations beyond the end of the testing phase.

 

Can I select which of my contacts will receive the subject line split test?

No, the test group will be randomly selected to send to 20% of your chosen recipients.

 

Can I cancel a split test after the test has started?

No, only our Support Team has the access to cancel a split test after it's started. You can, however, end the split test at any time during the 4-hour test period by manually selecting a winner.

 

Can I view my split test results in the response summary email?

No, you can't because split tests are not included in the response summary email that gets sent 24 hours after you send the campaigns, but you can see your results at any time by going to the Response section of your account.

 

Can I edit a split test if it's scheduled to send later?

Yes, you can, please refer to our instructions for editing a scheduled mailing. Just be aware that canceling a scheduled mailing reverts it to a saved draft, and this will deactivate your split test settings.

To re-add your split test settings, on the Review & Send page you'll need to click on the Switch to split test button again. Variation A will be preserved, but you'll need to re-enter the other variations of your subject line.

 

Will new contacts that are added after the test has started be sent the campaign?

No. Any contacts added to your recipient groups or segments while the test is in progress will have to be targeted separately.

Additionally, if you have scheduled the split test in advance, any contacts who were added to the groups / segments that make up the audience of the mailing after the mailing was scheduled will not receive it. This is a result of the way split tests are queued for sending and cannot be changed.

Additional Resources

• Campaigns Tab: Overview
• Creating a Content A/B Split Test
• Sending or Scheduling a Campaign

---

*In April 2022, Emma's legacy editor was replaced with a new drag and drop editor. Accounts that existed prior to this change still have access to the legacy editor through their previously saved templates and campaigns, however new, from-scratch campaigns can only be created in the new drag and drop editor. Accounts that were created after this change only have access to the new drag and drop editor. Please refer to our new editor FAQ article for more information.