Analyze Product Success, Not Failure

One of the most common fallacies when building a product is trying to fix failure by understanding failure. (I’m using product in the broad sense here where it could be a website, blog, app, game, e-commerce store and so on.)

Take some business metrics: acquisition, activation, conversion, and retention. Let’s say we want to improve them. It feels natural to start by:

1. Assessing why visitors who visit don’t sign up.
2. Analyzing why users who sign up don’t buy.
3. Understanding why customers who buy churn.

It makes intuitive sense to improve something bad. In engineering if you want to make your product have higher quality you fix the most common bugs. In machine learning if you want to improve the performance of your model you fix the worst cases. In college if you are struggling with your GPA you go to the office hours of the class you’re performing worst in.

The reason this doesn’t work when analyzing products is there are too many confounding variables. There are too many dimensions. Why someone who visited a website didn’t sign up might have nothing to do with your website. Or it might. You can’t tell without talking to them. That’s the problem. And the chances of them wanting to talk to you if they did not find what they wanted is low. (Even if you could pay them to talk to you, there’s so much nuance in the utility you could get, I’ll save it for another post.)

Maybe they thought your product was a meal planner to make sure they buy enough groceries for their family but it was actually a meal planner to get ripped so they clicked the back button. Maybe they saw your website as a scam and didn’t trust you with their information. Maybe they thought your product wasn’t going to help them achieve whatever outcome they thought it would at first. Maybe Pippeto the cat was missing for 3 days and just walked through the cat door as they got to your website. We don’t know. 🤷‍♂️

So what can you do to eliminate the uncertainty of whether or not the people you are analyzing wanted the outcome your product provides? You look at people who did perform your desired action such as buying. This eliminates the question of whether or not they had the intent to try your product. Very few people will accidentally buy and use your product for three days.

There’s a tremendous amount you can learn from people who did perform the action beneficial to your business and them. For example, you can go back in time and examine three simple things about people who purchased your product:

1. Where did they struggle to do something?
2. Where did they struggle to understand?
3. What did they see that catapulted them to buy?

Chances are, if John struggled for 30 minutes with widget D but ultimately figured it out and did purchase, there were 10X other people who did not. If Becky was feeling the solution might not work and felt weird about giving the website her credit card but took a risk, there were 10X other people who did not. If Larry saw the feedback poll about customer satisfaction buried in the blog somewhere and felt 40% of people succeeding with the product was very sincere which got him to purchase, 10X other people did not see that poll.

By understanding the most common places where customers who did purchase struggled to perform, understand, or simply discovered something convincing, you have the blueprint of improvements you can make. You also learn about who does want the outcome you provide and how they go about it. All because you eliminate the confounding factors and focus on understanding “high intent” visitors.