Product manager interviews usually include a section on metrics. As a data scientist at Uber, I’ve often given or helped friends prepare for these interviews. The difference between candidates who crush the metric questions and those who struggle turns, as far as I can tell, on whether they have a framework that they can apply.
Below is the framework I use for answering metric questions. This framework is primarily intended for questions where you are asked to define metrics for a new feature or product. Although there are a lot of points to hit below, don’t talk solidly for twenty minutes. Instead, ask questions as you go and respond to any pointers from the interviewer.
1. Understand the context and details 🔎
Make sure that you understand the context and key details of the problem before diving in. If there is a change to the product, get clear on what that change is and who it will impact. It is also helpful if you can get an idea of, or make some assumptions about, how you will measure the impact of the product. If you are tracking change over time, for example, you may end up choosing different metrics than if you have been given two weeks to run an experiment.
2. Define your goals 🏆
Probably the number one reason people mess up metric questions is that they don’t start by thinking through and defining the business goals. Without being clear on these goals, you’re just plucking metrics from the air and may miss the point entirely. Setting out the business goals also shows your interviewer that you are logically thinking through the problem.
Let’s suppose we are an eCommerce company planning to experiment with allowing customers to choose from multiple different colors of socks, rather than just one color. We may be doing this to increase sock sales, but it could instead be that we’re hoping to cross-sell interesting shirts to people buying funky socks. Or perhaps the new socks cost less than our current options. The possibilities are endless and it’s vital to be clear.
3. Choose a primary metric 📈
When it comes to choosing metrics, start with a metric that comes as close as possible to measure whether you have achieved your main business goal. In an experiment, this is the metric that you would use to make a stop-go decision. Often your primary metric follows so obviously from your main business goal that it can seem almost redundant to spell it out, but that’s fine. You’re going through the process of turning your main business goal into something that could actually be measured. If your goal is to increase sales of socks, for example, then a reasonable starting point for your metric would be the number of socks sold.
4. Unpack your primary metric 📦
It is important to define your metric with sufficient clarity that someone would have an idea of how to calculate it.
When using common metrics such as conversion or retention, it’s particularly crucial to make it clear that you have thought about what these terms actually mean in the context that you’re considering. What does a user have to do in order to have converted, for example, and over what time period will you look to see if they have done this?
There are two key points to consider:
- The time frame over which to measure your metric: Are you going to look at the number of socks sold in an hour, a week, or some other period? Consider the customer lifecycle in choosing your metric timespan, for example, people decide what socks to buy more quickly than choosing a home to purchase. You may need to make a tradeoff: while you would often like to understand customer behavior over a long period, this delays your results.
- Events you would need to define to calculate your metric: If we’re looking at stock sales over a week, does that week begin when a user hits the experiment, hits a particular page, or at some other point? Is a sale completed when a user clicks on the order button, when an order is shipped, or at some other point in time? There is no hard and fast rule about how precise to be in defining metrics. A good approach is to show that you are aware of the points that need clarifying and suggest that you can dig deeper if the interviewer wants to.
5. Briefly note the weaknesses of your metric 🤕
It’s worth showing that you are aware of your metric’s limitations. Don’t spend too long on this — a few sentences may well do it. If the timeframe used is not the one you really care about, for example, you can note this. You may choose to look at behavior over a week because you need results quickly when what you really care about is the value of the customer over their entire lifetime.
6. Select supporting metrics 🏋️
Select other metrics to support your primary metric, since anyone metric will be far from perfect, and to potentially shed more light on how customer behavior is being impacted. The limitations of your primary metric should give some ideas. You can also think through what in the customer journey you would expect to change if your primary metric is being impacted as you hypothesize. Considering more than one time period for a given metric can also be worthwhile.
You might hypothesize, for example, that more visitors will choose to buy a pair of socks if there are more color options available and we will, therefore, see an increase in the number of socks sold. In this case, it would be a good idea to measure the proportion of customers converting, as well as the total number of socks sold. You might also look over a longer period if you believe that the product change will impact longer-term behavior.
If your supporting metrics have strengths or limitations, you can briefly note these as you go.
7. Select defensive metrics ⛑
It is good to be a little pessimistic. Consider what could potentially go wrong with your new product or feature and how you will detect that. It’s often good to consider some measure of the number of complaints or queries that you’re receiving from customers (for example, # tickets/# sales) and at least one more direct measure of things breaking (for example, % of items returned). Again note the key details of these metrics, such as the time window in which you will measure if a sock is returned.
8. Summarize your metrics 📝
If you’ve touched on multiple metrics, as should be the case, it is good to recap, highlighting the most important metrics as you go and any particularly important strengths or limitations.
Health warning ⚠️ If you choose to apply this framework, do use your judgment in doing so. It is not the only way to approach metric questions, nor is it officially approved by any company I’ve worked for. What is expected of you will depend on the role and level that you apply to, while the clarity of questions, guidance that you receive, and the flow of the conversation will vary across interviewers.
That being said, I’ve used this framework in getting multiple data science roles and found that people following a similar approach tend to perform better in product manager interviews (and not just when I’m the interviewer), so I hope that it will be helpful in your job hunt. If you have any feedback or suggestions, do comment below or reach out on Twitter.