The Basics of Experimentation and Why It’s Key to Your Startup’s Growth

Some people say give the customers what they want, but that’s not my approach. Our job is to figure out what they’re going to want before they do. – Steve Jobs

Understanding customer preferences is critical in driving business growth, but customers typically don’t know what they want. As a business, it is beneficial to understand customers better and deduce what they want from you. This is where experiments help!

Startups with limited historical data typically get a better understanding of customer needs by running business experiments over a feature rather than from running analyses over historical data. You can make experimentation the key to growth for your startup if you can establish the right strategy, approach and methodologies. Let us first discuss what experimentation is.

Related: Experimentation and A/B Testing: A Must-Use E-commerce Growth Strategy

What is experimentation?

Experimentation, or what is sometimes loosely referred to as A/B testing, is a method where a business hypothesis is practically tested on consumers. Often an organization might not have historical data to analyze the business decisions. Similarly, they might be looking at some decisions where one cannot have historical data, like testing a new pricing strategy they have never tried. In these scenarios, we conduct an experiment.

We follow a methodology where we take a small sample from the entire population that will be affected by the decision and introduce them to a new feature. We compare the results observed from this test group to a control group that wasn’t introduced to the feature and understand if that particular feature can be beneficial to the consumers and the business. Using this specific methodology, we evaluate which of our hypotheses will be more valuable and implement the same. That’s experimentation in a nutshell for you.

This three-part series of articles will cover what experimentation is, why one should adopt it, elements of a successful experiment, the common reasons for experiment failure and some behavioral biases affecting experiment outcomes.

Related: Transform Your Business by Encouraging Experimentation and Change

Why experiment?

The two obvious reasons for conducting experiments are hypothesis testing and proving causality:

Hypothesis testing:

Typically, humans make decisions based on gut feelings and intuitions. Data analytics is an anti-thesis that supports data-based decision-making. But not all data are the same. You will find yourself in situations where you believe specific changes in the feature can increase your primary metric (such as growth or revenue). The hypothesis might sound reasonable to you and your colleagues, but it’s not guaranteed success as you do not have any backing data. In such a situation, experimentation is that friend who can provide you with a data-backed answer that can validate (or nullify) your hypothesis.

To prove causality:

Correlation vs. causality is a living issue in data analysis. Two or more variables are considered related in a statistical context if the values of one variable increase or decrease as the value of one variable changes. This change can have two cases:

Correlation is a statistical measure (expressed as a value between -1 and 1) that describes the magnitude and direction of a relationship between two or more variables. However, a correlation between variables does not automatically mean that the change in one variable is the cause of the difference in the value of the other variable.

Causation indicates that the change in one variable results from the changes of another variable, i.e., a cause-and-effect relationship exists between the two variables.

Theoretically, the difference between correlation and causation is easy to identify. However, it does not remain easy in practice. Randomized experiments help differentiate between these two realities to find truly causal effects. Randomized experiments are the norm in the real world to understand if a specific change can create a difference in the outcome. For example, a randomized controlled clinical experiment establishing a pill’s effectiveness helps to confirm that the effect is a result of the intervention and not anything else.

Not resource-intensive like real-world experiments:

Digital experiments are not resource-intensive as compared to offline experiments. It doesn’t need any additional funding or arrangements required for real-world experiments. You don’t need to recruit participants or tell users they are part of an experiment! So, how exactly is it different from data analytics?

Related: To Stay Successful, You Must Continuously Evolve Your Business

What makes experiments different from analytics?

The data source for the analysis is the fundamental element differentiating experimentation from analytics. Typically, there are two ways to get data for quantitative analysis:

Historical: Historical data includes data stored by the company in their data warehouses about what has happened in the past, which helps understand how users behave on your platform. Historical data helps run various analyses, including user behavior and identifying customer segments.

Experimental: Experiments help you validate business hypotheses as a new change idea will not have essential historical data to validate the change. One could conduct experiments to observe user responses to an app change or feature addition and compare that to the control group’s behavior.

Experimentation can be your friend and a business enabler, which is a widely discussed and commonly used process but not usually executed without fallacies. The next post in this series will discuss the key elements that define a successful experiment and the four common reasons for experiment failures.