Sensitivity Analysis

Also called Power Analysis, this technique helps you determine how many observations are needed for the results of A/B Testing or Multi-armed Bandit to be considered reliable.

To use the method, you need to provide two input parameters:

• The size of the change you expect to measure, for e.g., 5% increase in CTR
• The numerical confidence you want in your results, for e.g., 95%

You’ll typically need fairly large samples to get trustworthy results.