A/B Testing

Slides

What is A/B Testing?
- Break users into randomly assigned groups
- Show each two SMALL variations (for features, not for entire sites)
- Optimization of features, not usually for huge changes
- Must have a clear Key Performance Indicator (KPI)
- Must have single measurable action
Benefits
- Measures real user behavior, instead of relying on your flawed insights
- Can measure really small differences that can have big impact
  - Amazon bigger button, 1% increase, so conversion rate goes from 2% to 2.02%. Worth it?
  - That's $69 million / year
- Resolve tradeoffs when there's conflict
  - Coupon vs not? Usually 25-50% more sales when NO coupon is required. Doesn't necessarily apply to all sites. If you make more money with the coupon, use that
  o Cheap
Downsides
- Basically no insight into why
- Not generalizable (a bigger button may work on your site, but that's not evidence it will work on all sites)
Statistical significance
- How many people to you need to find a significant difference at a given % difference?
In practice
- Make sure people remain in the same group over the period of the test (cookies)
- Start with a small set and work up to 50/50
- Have an abort procedure if there are really bad implications from the change

What do you want to test?
What will you measure
How will you split it
Are there ethical concerns? Policy concerns?
How many people do you expect will participate / what percentage? How many do you need to reach statistical significance? (use table)