Experiment-driven Development 

Experiment-driven development entails running tiny tests to evaluate options/ideas that you’ve in an iterative and structured fashion to incrementally achieve results.

Why it matters

Strategically

• Always scale after achieving fitment. Optimizing too early stunts iterative discovery.
• Incrementally discover and improve towards a larger goal.

Operationally

• Speeds up discovery. Short iterative experiments will pop-up the unknowns. Additionally, the process forces you to stop and reflect.
• Encourages risk-taking: An experiment might fail to achieve what we set out to do, but it succeeded in teaching us something new about audience. Additionally, each experiment is tiny enough that it does not result in outright failure of the product.
• Helps convince. Gives you ammunition to convince others with evidence.
• Helps seek permission. It comes from the vantage point that you don’t know and hence you will experiment. This helps bypass egos and strongly held positions. Additionally, each step is tiny so its easy to convince stakeholders to try it out.
• Promotes customer-centricity: Each experiment’s output is based on behavioral analytics.

How

• Build a culture of asking questions and proposing explanations as hypothesis based on the limited information available.

• Estimate data for each of the hypothesis using the RICE model.
  • Reach: How many users will use it?
  • Impact: What outcomes (business metrics) will it grow and by how much?
  • Confidence: Given your capability and constraints, how difficult would it be to test this
  • Effort: What would it take to implement this
• Prioritize which hypothesis to focus on
• Design your tests
  • Define context: What do you know? What do you have access to? What do you think is going on? What do you want to find out?
  • Summarize prior research: How have you experimented in this area before? What did you learn from your results and analysis? What further research do you need to do before you start the experiment?
  • Obstacles and risks: What are the risks in the following areas: Business, Usability, Technical? How will this affect? How will you mitigate these results?
  • Success: What is the primary success metric? What is the secondary metrics that this experiment will impact? What is the benchmark today? How much do you expect it to change?
  • Who and how many: Who will you test this on? Use Sensitivity Analysis to identify how large should be the sample size.
  • When: When will you conduct this test? How long will it last?
• Conduct the tests
  • Do anyone need it? Qualitative Research, Interviews, Paper Sketch.
  • They need it. Would anyone use it? Build an unscalable version, for example, prototype in Figma.
  • Does a significantly large audience need it? Digital advertisement campaign with a landing page
  • How do we build the product properly? A/B Testing, Multi-armed Bandit