Configure an A/B Test to Compare Promotions

Review which promotions perform the best against a control group, compare them by using A/B tests.

1. From App Launcher select Merchant Tools.
2. Expand Online Marketing | Promotions.
3. Make sure that the three promotions exist. Add any that are missing.
4. Select A/B Tests, and create a test with the ID Content Promotion Test.
   1. Enable the test and enter an optional description that is for usage in the testing.
   2. Include an email address for a notification when there are statistically significant results for the key metric.
   3. Select Orders as the key metric, but consider other metrics when looking at the results.
      The test is looking for the promotion that leads to the highest Conversion Rate. The test is also looking for the one that maximizes Average Order Value and minimizes the Average Discount Amount.
5. Select per customer or per browser to provide more control.
   If customers return to the site in a repeat visit during the test, you want to offer them the same promotion.

   • Per customer tracks in the customer profile, so anytime the customer is recognized they participate in the same test segment.
   • Per browser tracks similarly to per customer except the participation is connected to the browser history and cookies.
6. Configure the test to include participants from the Everyone group, so that all customers can participate.
7. Set up three test segments.
   • Free Shipping over 100 allocated 30%
   • 10% off 100 allocated 30%
   • $25 off 200 allocated 30%

   

   Note Business Manager automatically allocates the remaining 10% to the control group.
8. Select Content Slots and identify the slot Header Banner on the site.
9. Show a banner for each included promotion, and create three slot configurations for, each with the content for one of the three promotions.
10. Return to the test and include the corresponding slot configuration as an experience for each test group, in addition to the control.
11. Set the test start date to midnight and the end date to 9PM Saturday.
    You want an answer before the summer clearance begins next Monday.
12. After creating the test on staging, replicate it to production.
    The test goes into effect at midnight, and you receive an email alert when there are results.
13. Review the results page.
    • Orders. The Free Shipping over $100 segment shows a statistically significant difference in Order Rate at 5%. The other segments show a roughly equal Order Rate of 4%.
    • Average Order Value. The $25 off $200 segment has a statistically significant difference at $59.50. The other segments show a roughly equal Average Order Value around $47.
    • Average Revenue per Visit. The Free Shipping over 100 segment is higher, but the difference isn't statistically significant.