Experiment-driven Development (EDD)

In the real world, attributing an outcome to a specific action is very hard.

Unlike A/B Testing, multi-arm helps you test multiple options on one aspect of your product or service at the same time. Additionally, it also takes the action of scaling the best performing option by itself.

How it works: Let’s explore one of the methods: Epsilon-Greedy.

• On 20% of your customers, it explores or tests out multiple options to see which one performs best. On 80% of your customers, it exploits or adopts at scale the options that has the best performance.
• This mode kicks in once you’ve at least one decent option. Prior to it, 100% of the customers are on explore mode.

The details: You’ve to choose interchangeable options. For example, you can use it for posts headlines or for cover image in a post. However, you cannot use it for things like different pricing tiers.

A/B Testing lets you test two different options on one aspect of your product or service at the same time in production to determine which one is performing better.

How it works: It uses segmentation to divider the selected user base into two buckets. The Experimental Group sees the change, while the the Control Group doesn’t. The outcome of the A/B test is determined by measuring the difference in performance of both the groups.

On the downside: A/B tests can test only two options. Hence, if you’ve multiple options to choose from, then you can end up wasting a lot of time determining the best performing option. That’s where multi-armed bandit testing comes in.

The components of decision-making are:

• What monitors areas like shipments, sales, and market share to keep an eye on the key metrics for the business.
• When exceptions occur
• Why drills down to the drivers—country, territory, product line, and store levels—to understand the specifics behind what is happening.
• Levers that you can pull, such as pricing, advertising, and product mix
• Estimates of what that action can potentially deliver.

The New York Times identifies engagement moments in a reader’s routine so that they become part of the reader’s day-to-day life, something that they cannot live without. For example, they know that their podcast listeners are most attentive during “yellow moments.”

Source: reutersinstitute.politics.ox.ac.uk/sites/default/files/2022-01/Newman – Trends and Predictions 2022 FINAL.pdf

Build a culture of asking questions and proposing proposed explanation based on the limited information available before delving into complex problems.

Here’s how you write: “If we change… (action), then we will see… (result), because… (clarify rationale).

The details: Hypotheses are generated by human intuition based on the collective intelligence and experience of stakeholders and their understanding of the business and their environment. This is often generated by asking questions:

• What are you trying to do?
• What is happening that is not supposed to happen?
• What could drive it?
• Why is it happening?
• What is happening?
• Why is it happening?
• What are your hunches?
• What opportunity are you exploring?
• What problem are you looking to solve?
• Are their gaps in your knowledge that you are trying to fill-in?