A/B Testing

A/B Testing lets you test two different options on one aspect of your product or service at the same time in production to determine which one is performing better.

How it works: Always start with a hypothesis. Use segmentation to divide the selected user base into two buckets. The Experimental or Treatment Group sees the change, while the the Control Group doesn’t. The outcome of the A/B test is determined by measuring the difference in performance of both the groups.

On the downside: A/B tests can test only two options. Hence, if you’ve multiple options to choose from, then you can end up wasting a lot of time determining the best performing option. That’s where multi-armed bandit testing comes in.

Related: Experiment-driven Development