Cohort Analysis: The Key to Data-Driven Product Decisions

Your introduction to cohort analysis. Discover how they can help you make the right business decisions and drive forward product success.

Paula Becchetti
17 min read

In the world of SaaS analytics, it’s oh-so-easy to get lost in a pile of data. We keep telling ourselves that our product’s long-term success depends a lot on the depth of our consumer behavior analysis, but when we have to dive into the data, most of our analytical efforts stop at downloads or Monthly Active Users (MAUs) measurements—both of which are superficial ways of assessing your product’s growth

We often forget that these metrics don’t answer the most basic and valuable questions around user engagement, metrics such as who is repeatedly visiting our app or why and when do users lose their interest? If you really want to get insightful data to sharpen your product’s value proposition, cohort analysis is your best friend.

What is cohort analysis? #

Cohort analysis is a type of analytics that helps measure user engagement over a period. Its fundamental characteristic is the way data is selected and organized in different groups of users, the cohorts, each one defined by a set of shared attributes and experiences within a defined time-span. So, instead of looking at all users as a single shapeless mass, you’re able to break the analysis into more coherent batches to better understand your consumers’ actions and make informed product decisions to increase your customer retention rate and revenue. 

As its name states, this method is based on the examination of cohorts’ performance over time. You’ll probably have heard this word before in the academic world, where a cohort is a group of students that graduated in the same year. And here comes the crux of the matter: cohort members can’t be segregated by any given characteristic, they have to be bound by a common event that occurred during a specific period in time. 

A good example of a cohort in business analysis is a group of users that started to interact with an app the same week or month or entered your website for the first time on the same day—you could think of them as the “class of week 36”.

Another relevant note about cohort analysis is that it gives you insights into tendencies, not causes. It won’t point out the exact source of the ups and downs of your customer retention metrics, but it will be of invaluable help for you to understand why your customers are engaging with your offering (or not) and test different campaigns and strategies in the road to find the best value proposition for your product. 

Why you should be using cohort analysis #

Cohort analysis is imperative for product-led growth—it allows you to examine trends over time and measure different groups of users—you can see how they accept your product. This powerful method shows how new and old users relate to your product and lets you recognize and understand their behavioral changes in the face of engagement marketing strategies such as:

  • Ad content

  • Promotional campaigns

  • Website redesigns

  • New product lines

  • Service offerings and discounts

Among an almost infinite amount of possibilities. Here are some of the biggest benefits that come with cohort analysis:

Differentiate growth and engagement metrics #

Customer engagement is commonly mistaken for growth. Cohort analysis helps you recognize the difference. It’s easy to assume that your customers are loving your product when figures like MAUs go up, but you can be on the wrong track if you get blinded by momentary peaks of growth metrics. 

In cases like this, cohort analysis shows you how many of those acquisitions are turning into loyal customers, who will continue buying, and if channels with high acquisition numbers actually translate into bigger profits. Shiny numbers can be masking future growth and sustainability issues. 

Avoid vanity metrics #

Cohort analysis prevents you from the problems that come with vanity metrics. These are metrics that make you feel and look good but those that can influence you to make harmful business decisions, because deep down, they do not mean a thing without additional context. In other words, cohort analysis protects your business from your ego. 

When we dive into Google Analytics, that increase in new sessions on our website makes us feel great. But, is this really a good omen or a misleading diversion from that decreasing retention rate? Exercise the muscle of cohort analysis, including cohort analysis filters in your Google Analytics report to get solid, actionable data. 

Understand the whys behind the whats #

With customer cohort analysis you can go one step further in knowing which customers churn, which ones stay, and when they do it. You can understand the reasons behind users’ movements.

Comparing variables is an extremely powerful thing because it brings you a wider perspective of how metrics are interrelated—beyond descriptions and into solutions. For example, you can see the number of visits to your website had an exceptional peak during a week your brand got featured in a US publication simultaneously with the launching of your redesigned newsletter. Analyzing different cohorts during that week, groups from different countries and channel origins, can crack the mystery and bring new insights into your most valuable strategy.  

Exercise a long term vision #

Businesses need time to thrive and keep up with the competition. That’s why cohort analysis is so valuable. With cohorts, you can analyze and understand the lifecycles from both users and your products—as well as getting a comprehensive look at seasonality. You can gather knowledge on how new users respond in comparison to previous ones—and if they bring more or less revenue, how seasons impact on every cohort purchase habits, and if users acquired during a special occasion (e.g. Black Friday or Christmas) behave differently than others acquire in other times. 

Getting to know the long term value of your users helps you take care of the future health of your business and incorporate data-driven expectations into your forecasts to make them more accurate. 

Identify user flows  #

Monitoring different types of users, helps product teams understand stage-specific needs and identify issues sooner. New users will probably need more guidance and better user onboarding, advanced users will want new feature announcements and product upgrades—and all of them can experience the same or different problems along the way. 

Cohort analysis reveals where you’ll need to offer extra help, like product tours to guide users or new features to keep the loyalty levels high, and when disturbing trends start to appear in a cohort, so you can react quickly to avoid its spread on the rest of the user base. 

How to read cohort analysis charts #

After walking through the cohort analysis definition and its multiple benefits for your business, it’s time to take a closer look at what a cohort analysis looks like. 

The data used to carry on a cohort analysis is arranged in a chart like below, called a “cohort chart”, where the combined metric you’re analyzing is displayed in the intersections between the vertical axis and the horizontal one:

(Source)

At a glance it can seem overwhelming, so let’s go through it together:

  1. Take a look at the vertical axis. The cohorts generally run in this direction, with the oldest cohorts on the top and the newest at the bottom. In this particular chart, we have monthly cohorts of people who installed an app, starting from February through to August. The first column shows us the dates of installation, the second one the number of users who installed the app that month. 

  2. The horizontal axis shows the passage of time. Across it, you’ll find different periods (always from the same length) since the start of each cohort. In the example, they range from month 0 (the month of acquisition) to month 6, six months from the month of acquisition. 

  3. Then comes the core data, placed in the cells in the middle, where stands the metric you’re examining—in this case, retention rates N months after acquisition. You’ll always have more data from the oldest cohorts because they were analyzed earlier, resulting in the typical triangle shape of a cohort chart—therefore sometimes referred to as the triangle chart. 

  4. After visualizing the whole chart, there are three ways to read it. 

    1. Vertically. If we go across a single column, we can analyze the progress of the metric in the same month of acquisition (same moment in the user life cycle) across cohorts. 

    2. Horizontally. If we look at the same row, we’ll be analyzing the evolution of the metric in the same cohort throughout time. In the example, how retention rates decrease from month 0 to month 6 in the first cohort. 

    3. Diagonally. We can take a peek at how the metric looks like the same calendar month in all cohorts—for example, August retention rates.  

A few additional notes on cohort chart reading. It’s important to be cautious about the size of the cohorts. If they’re too small, metrics will vary widely and potential analyses will be worthless. A good practice when it comes to visualizing trends is to use color shading. It will be easier to see how the value progress or decays, and when anomalies appear. 

Acquisition and behavioral cohort analyses #

Every cohort analysis starts with its cohort definition. The nature of the traits selected to divide users into groups will impact the approach, the type of questions we can answer, and the strategic value of the answers we can come up with. In this sense, we can recognize two main types of cohorts which will determine the class of analysis we can conduct: acquisition and behavioral. 

Acquisition cohorts, as its name suggests, are based on when members sign up for a product. Like any cohort, as mentioned earlier, groups are defined by a shared moment and a common event, but that event can’t be anything other than acquisition (purchase, registration, download, you name it). Acquisition cohorts are great to identify when new users are churning, hence measuring retention and churn rates across a period. 

Analyses like this are useful, for example, to measure the success of an app launching. You can break down your cohorts by day for the first week of launching, by week during the first month, and by month during the first 6 months after your product hits the market to see how long people continue to use your app from their starting point. Look out for cells where retention rates drop off drastically—is it on day 1, when they had to register, or is it during week 2 when they finished your onboarding material? That way you’ll get a sense of where potential issues are located. 

There are some clear limits to what you can do with an acquisition cohort analysis, especially when you want to dive into the why of people leaving or staying—and that’s when behavioral cohorts come to the rescue. 

With behavioral cohorts, the event that defines the group can be any discrete action performed with the product after initial acquisition. Users exhibit hundreds of small behaviors that eventually influence their final decision of believing or not in the product: using a certain feature, enabling notifications, integrating with other applications, uploading personal information, etc. 

As previously mentioned, cohort analysis won’t point out the exact causes of retention and churn, but with behavioral cohort analysis, you can test common behaviors of your most engaged users and find sticky parts of your app. Similarly, you can track cohorts during a certain period to see how long they stay active after they interact with a tricky feature of your app. 

For example, if you’re doubting your onboarding UX, you can perform an A/B test offering different onboarding experiences to different users and then tracking their responses within the first two weeks as separate cohorts. That way, you’ll be able to compare retention rates and see if one option is much more successful at engaging your customers. 

💡 Chameleon Tip: How to get started with cohort analysis #

Now that we’ve gone through the basics of cohort analysis, it’s time to get to business. With Chameleon, you can effortlessly collect all your product data in an easy-to-read dashboard, thanks to our seamless integrations with Google Analytics, Mixpanel, and Amplitude. Once you’re there, all that’s left to do is experiment with your product growth, learn from the compared information you can get from each cohort analysis, and keep moving in a virtuous cycle of improvement. 

Discover the Mixpanel and Chameleon integration

Leverage your Mixpanel cohorts and target them with specific in-product Chameleon Experiences.

One tip to squeeze the most out of your cohort analysis—don’t forget to use annotations. If you track your activities such as marketing campaigns or product improvements within a calendar and later compare them with your cohort charts, you’ll be able to relate important movements in your metrics on any given period or cohort with potential sources of impact. For instance, if you send newsletters on Tuesdays, and your Tuesday cohorts have higher retention rates, perhaps that activity is worth keeping and improving. 

And last but not least, put your experimentation hat on and have fun. Test new design hypotheses and later look at your cohort charts to check results. Try out different buyer personas each month to find out your most engaged audience. Run A/B test modifications on channels, campaigns, or new product offerings, and let your cohort analysis tell you what works and what doesn’t. Cohort analysis is your loyal friend for all data-driven, customer-centric decisions. 

What are your thoughts?

You might also be interested in...