Salesforce Docs Portal

Configure an A/B Test to Compare Search Order

Test which of two search order rules produce the highest revenue for a category. Use active data Top Sellers and Best Margin to define two new sorting rules.

  1. Click App Launcher App Launcher, select Merchant Tools, and then go to Online Marketing | A/B Tests.
  2. Configure an A/B Test to compare search and sorting.
  3. Enable the test and enter an optional description for it.
  4. Include an email address for notifications when there are statistically significant results for the key metric.
  5. Select Average Order Value as the key metric.
  6. Configure participation to expire per Session.
  7. Set the test to run from August 2 through August 31.
  8. Make the test available to all customers via the Everyone group.
  9. Create two test segments:
    • Sets a sorting rule test segment to Top Sellers as the default for a search refined by that category, allocated to 50% of the participants.
    • Sets a sorting rule test segment to Best Margin as the default for a search refined by that category, allocated to 45% of the participants.
    • The control group gets a different rule, Fastest Sellers, because it's configured site-wide as the default for that category. This is allocated to 5%.

You replicate the test in July, and it takes effect on August 2. You check the results occasionally, but there's no clear conclusion. The test ends on August 31, and you check the results for a final time. There were minimal differences among the groups for some metrics, including Average Order Value but even with 2,500 orders, none of these differences were statistically significant.

Redesign the Top Sellers and Best Margin rules to use Sales Velocity for the last 7 days, instead of the last 365 days. This change might show whether the sort order is more responsive to current trends. Copy the test and run it again for the 30 days after Labor Day. The changes to the sorting rules themselves are enough to affect the test. It's not necessary to change the test groups. Replicate the changed sorting rules and the copied A/B test to production.

Two weeks later, an email indicates that the Top Sellers rule shows a statistically higher Average Order Value. The timeliness of the active data was the key difference. Select the Top Sellers rule as the default.