A/B testing email content and subject lines

What it does

An A/B test splits a journey's audience across two versions of a message — different subject line, different content, or both — measures which performs better on a metric you choose, and then sends the winner to everyone else. The winner is declared on statistical significance, not on whichever number happens to be higher.

Key facts

The test is a journey tile, named Test which variation performs better. You're testing message versions inside a journey, not standalone assets.
Two versions only. There is no multivariate option and no third arm.
Winning metric is one of three: journey goal events hit, clicks, or opens.
Experiment distribution defaults to a 50-50 split, and the slider allows anywhere from 10% to 90% per version.
In segment-based journeys you can also hold back an Initial Audience percentage — test on a slice, send the winner to the rest. Trigger-based journeys don't have a fixed audience to hold back from, so this doesn't apply.
The control group option isn't available on a segment-based journey that contains attribute branches or tiles that change the segment total, because the system can't predict the holdback size.
Automatic ending: the test stops when significance is reached. If nothing is significant after 30 days, the Default version is sent. You must nominate a default version.
Manual ending: you can stop a test early and choose which version goes out. Releasing the held-back group takes up to four hours.
Microsoft's own guidance is at least five hours between start and end for a result worth reading, and the background evaluation itself can take several hours.
Audience size matters more than people want it to. Under 1,000 people the split gets uneven and the result gets noisy; under 10 members the groups are essentially arbitrary.
Test states are Draft (editable), In progress (settings locked), Stopped (you pick the version) and Ended. An ended test can't be reused — you build a new one.
Results appear in the journey's analytics and on the goal analytics dashboard, with a winner badge on the version that took it.

When to use / skip

Use it where the audience is large, the send is recurring, and someone will actually act on the answer. A monthly newsletter to 200,000 contacts is a good candidate. A quarterly event invite to 400 people is not — you will get a result, it will not mean anything, and the client will build a content strategy on it.

Skip it entirely on the first release unless the client has asked for it. Most implementations have bigger problems than tuning a subject line, and an A/B test on a journey that isn't yet trusted just adds a variable to every "why did this person get that email" conversation.

Where it genuinely pays off is subject line testing on high-volume, high-frequency sends, because that's the one variable with enough signal to move a metric within the window the test allows.

Configuration decisions

Which metric to win on. Opens are the weakest signal now that mail privacy protection inflates them; clicks are more honest; goal events are the only one tied to business outcome and the only one that needs a journey goal configured first.
Split percentage, and whether to hold back an initial audience. A 10/10/80 shape gives you a clean winner-to-majority send; a 50-50 with no holdback gives you a faster read on a smaller total.
Automatic significance versus a fixed manual stop. Automatic is more defensible; manual fits a campaign with a hard deadline.
The default version, chosen deliberately as the safe option rather than whichever was built first.
What is actually being varied — subject line alone is a clean experiment; subject and content together tells you nothing about which one moved the needle.
Whether the journey's structure allows a control group at all, which you need to know before designing the segment journey rather than after.

Gotchas

The 30-day timeout sends the default version. On a journey with a fixed campaign window, the test can quietly expire and everyone gets the default without anyone noticing there was a test.
Stopping a test doesn't release the audience immediately — up to four hours. Marketers watching the numbers assume it's broken.
Settings lock once the test is in progress. A typo in version B is a rebuild.
Ended tests can't be re-run. Recurring campaigns need a new test each cycle, which people forget when they clone a journey.
Small audiences produce a confident-looking winner badge on a difference that isn't real. The statistical guard helps, but it can't create signal that isn't there.
Adding attribute branches to a segment journey after the test is configured can remove the control group option, and the failure surfaces as a validation error at go-live rather than at design time.

Consultant notes

Ask what decision the client will make with the result before building anything. If the answer is "we'll see", the test is decoration.
Demo it on a segment big enough to conclude, or don't demo it. A demo test that ends inconclusive undersells a feature that works fine.
Tell them up front that opens are a degraded metric and steer them to clicks or goal events. This is a conversation worth having once, properly.
Check before go-live that a default version is set on every test, and that it's the one the brand would be happy to send to the whole list unattended.
Push back on testing content and subject together in the same experiment — it's the most common request and it produces an unusable answer.

Worth another look if more than two variants become supported, or if the significance window or 30-day timeout changes.