Understanding A/B Test Results in B2C Commerce

After you draw conclusions from a B2C Commerce A/B test, deploy some or all the experiences for its test groups to the site.

Typically, you want to deploy the experiences of the winning test group, but sometimes you want to deploy experiences for multiple groups. To deploy these experiences manually, create a campaign or add to an existing one. Before drawing conclusions, however, determine how long tests run, when they end, and how you know that the tests have run long enough. Key concepts to understand are confidence level and statistical significance.

Note Salesforce B2C Commerce doesn't update A/B test statistics for orders that originally failed and that you later manually open in Business Manager.

Statistical Significance

The point at which a test reaches statistical significance depends on multiple parameters. Typically the more similar your test subjects, the more metrics you test. The lower your traffic, the longer it takes to reach statistical significance. But what is statistical significance?

In statistics, statisticians consider a result statistically significant if it's unlikely to have occurred by chance. For example, you run an A/B test that includes 500 customers. The test showed that the average order value for September was 30% higher with an autumn-colored banner than with a dark-colored banner. Though the results can be statistically significant, is the difference important? Tests of significance include effect-size statistics, which approximate the size and thus the practical importance of the difference. Statisticians call the amount of evidence required to accept that an event is unlikely to have arisen by chance the confidence level.

Compare the metrics and values of the control and test groups on the A/B Testing page. B2C Commerce calculates the confidence level, which indicates the likelihood that these differences result from your change in site experience and not from random chance. When the confidence level reaches 90%, the result is statistically significant.

Note The statistical significance testing procedure assumes that you set the sample size before you start the test. Therefore, your initial results can vary until you reach the sample size. Remember this when you receive an email notifying you that your confidence level reaches 90%.

In the banner-color example, if the B2C Commerce-computed confidence level reaches 90%, consider the test result statistically significant. The merchandising team can use an autumn-colored banner in September, with a high degree of confidence that it drives better results than a dark-colored banner.

Test Length

How long to run a test depends on your average number of daily visitors, the percent of visitors included in a test, and other external factors. In general, run it until the test reaches statistical significance or until it's clear that it won't. Understand, though, that a B2C Commerce A/B test can run for up to 90 days.

An A/B test automatically ends on its end date unless a user disables it. The system sends an email to recipients configured on the A/B test if it reaches a 95% confidence level for its key metric. The test continues to run after the system sends the email, so the confidence level can dip below 95% after the email sends. The email sends only once to prevent duplicate emails if the confidence level changes from 95% to 94% and then back to over 95%. Deploy segment experiences even if the test doesn't reach a 95% confidence level. For example, deploy for a confidence level of 90%, 85%, and so on.