1

We are currently having an interresting discusison about a recent A/B Test we ran and i'd very much appreciate your thoughts on it:

We recently ran a A/B/C Test regarding the pricing of same day delivery:

  • Control, current state:: free same day delivery
  • Variation 1: 3 Dollars for same day delivery
  • Variation 2: 5 Dollars for same day delivery

We measured:

  • CR_Checkout: Overall Conversion Rate in the Checkout (including other delivery options) as: '# users in checkout / # transactions
  • CR_SameDay: Usage Rate same day delivery as: '# transactions with same day option / # transactions

Results compared to Control:

  • Variation 1:
  • Change in CR_Checkout_1: Not Significant
  • Change in CR_SameDay:_1 Significant decrease
  • Variation 2:
  • Change in CR_Checkout_2: Significant decrease
  • Change in CR_Sameday_2: Significant decrease

Question: Under the given circumstances - is it statistically sound to go on and make a profit calculation to compare Variation 1 and Variation 2, using the respective conversion rates, average order value (generic for both variants) and revenue from same day delivery fee - even if we have not proven a statistically significant difference between Variation 1 and Variation 2?

Example: Variation 1: 100 Users in Checkout x CR_Checkout_1 x Average Order Value + Revenue SameDayFee = 100 Variation 2: 100 Users in Checkout x CR_Checkout_2 x Average Order Value + Revenue SameDayFee = 105 Therefore Variation 2 should be implemented.

kjetil b halvorsen
  • 77,844
Sandro
  • 11
  • 1
    Welcome to CV.SE, what is your original hypothesis before you set up this test? – B.Liu Feb 05 '22 at 14:08
  • What is preventing you from tossing out the "control" results and performing your statistical tests between variation 1 vs variation 2? – Dave2e Feb 05 '22 at 14:39

0 Answers0