Online experiments are powerful tools to see the causal impact of a business strategy. Tweak something, see what users do.
The limitations of online experiments is that once you manipulate your user experience there’s no going back. Users are forever impacted by your choice.
If you give out a credit card with $100, $300, and $500 lines, people are going to spend more on the $500 line. They might also default more because they’re less able to pay back $500 than they are $100. As a result, you may lose these $500 users forever and might impact other aspects of their life, not to mention impact the marketing department’s strategy.
A group of users is like a single living organism, if you treat it like that. If you only measure one metric (conversion rate) on the entire massive audience, then you’ll optimize for the “average” user.
But nobody is average. We are all distinct. We fall into categories, sure. But to really optimize the whole business we need to optimize for pockets of users. Or, we just need to set up a system that’s personalized.
Takeaway? Personalize your experiments where possible. Consider the ethical and downstream impacts of the experiment.
_________________________
Bryan lives somewhere at the intersection of faith, fatherhood, and futurism and writes about tech, books, Christianity, gratitude, and whatever’s on his mind. If you liked reading, perhaps you’ll also like subscribing: