Cohort Analysis: Metrics for a Scalable Business

Discuss on Hacker News

Two companies

Alpha Labs: total signups after 6 weeks: 1995

Beta Works: total signups after 6 weeks: 1995

The “up and to the right graph” is identical for both of these companies. This is the image they show to their investors and advisors to get people excited.

It looks like they’ve both got good traction. So which do you invest in? Can we dig a little deeper?

Well what if I told you that Alpha Labs has 242 active users and Beta Works only has 26 active users? Now it seems obvious which is doing better. Where did Beta Works go wrong?

The problem

While both companies invested in Adwords, Content Marketing, and Social Media, Alpha Labs made sure their conversion rates remained constant as they scaled. This means that as they added new features to their product and improved it they constantly conducted cohort analysis to make sure that the same proportion of users weekly were being converted at each stage of their funnel. 

This is a critical aspect of building a product. If your conversion rates drop as you scale up, something isn’t correctly automated. 

Scalable: A healthy cohort report

So boring

This chart looks a lot more boring than the first one. There is no “up and to the right”. In fact, it almost looks like things have plateaued. This is the strength of a cohort report. It’s deceptively simple.

Every product has a lifecycle, it’s probably something like ours, but note that each state is mutually exclusive, a user can only be in a single state at a time.

How to read it

Each point on the X axis represents a group of users who signed up during that week. So 9/15/2012 shows all of the users who signed up during that week. The Y axis represents the percentage of users inside a cohort that are in a given state. 

I’m sure by now you’ve guess why this chart is so valuable. It removes all the bias of a growing user base and solely focuses on how many users per cohort you’re converting to the next stage of the funnel. This allows the product team to really focus on making each feature in the app actually drive these metrics. 

Unscalable: A sick cohort report

It’s pretty plain to see what went wrong. Early on Beta Works was virtually identical in conversion rates to Alpha Labs but as they started to pour on the advertising dollars they couldn’t keep up with the demand. The proportion of users at each stage of the funnel declined. 

How do we avoid this?

Declining conversion rates is a sign of poor scalability. Immediately throttle back your marketing spend and focus on improving your product.

Thank you

Feel free to take a look at how these calculations were done via this google doc spreadsheet.

-Cameron Westland