Downstream Impact

When you do an online experiment, you’re changing the future forever.

data science
experimentation
Author

Bryan Whiting

Published

June 22, 2022

Every pea is unique. But it’s still a pea.

Online experiments are powerful tools to see the causal impact of a business strategy. Tweak something, see what users do.

The limitations of online experiments is that once you manipulate your user experience there’s no going back. Users are forever impacted by your choice.

If you give out a credit card with $100, $300, and $500 lines, people are going to spend more on the $500 line. They might also default more because they’re less able to pay back $500 than they are $100. As a result, you may lose these $500 users forever and might impact other aspects of their life, not to mention impact the marketing department’s strategy.

A group of users is like a single living organism, if you treat it like that. If you only measure one metric (conversion rate) on the entire massive audience, then you’ll optimize for the “average” user.

But nobody is average. We are all distinct. We fall into categories, sure. But to really optimize the whole business we need to optimize for pockets of users. Or, we just need to set up a system that’s personalized.

Takeaway? Personalize your experiments where possible. Consider the ethical and downstream impacts of the experiment.