4 Lessons I Picked Up as a Machine Learning Product Manager

Following up on my first post about the role of a Machine Learning Product Manager, I would like to offer some of the key lessons my 10-year experience in the field taught me. I hope that they prove helpful for other PMs who are already in the field or are considering it as a career. 

Arthur C. Clarke is hard to argue with, especially when we look at the latest tech advancements such as ChatGPT and DALL-E 2.  This certainly rings true for our experience with ML/AI as end users; yet, it feels very different in the trenches of ML development where you quickly learn that ML is hard work and perseverance or if you wish, an “art and science”. ML requires effort, skill, and many iterations before the “magic” occurs. 

But why go through all that trouble if it's so painstaking? Well, simply because when ML does work, it is, indeed, nothing short of magical. The quality of well-tuned ML solutions - their accuracy, adaptability, and scalability leading to delightful, personalized, serendipitous experiences for every customer - is unparalleled. These kinds of experiences make products “sticky” and help businesses thrive by attracting and retaining more and more customers.

In non-customer-facing aspects of business, e.g. back office operations, employee tools, or infrastructure and equipment maintenance, ML drives productivity and efficiency through the automation of routine activities and through the ability to predict and thus prevent issues (so-called predictive maintenance). The impact of ML on the bottom line is what has powered the growing investment in it over the last decade. 

But this is a business perspective on ML. What about the individual motivation of PMs who work in this field? Personally, two things drive me: my genuine interest in the technology, and the high demand for this expertise which is good for job security. However attractive this job may seem, it is challenging and, I believe, hard to do for long without feeling strongly about the problem space and the people who work in it, ML engineers and data scientists. These are a special crowd of highly intelligent, skilled, and inquisitive people, so if they are your crowd, it helps persevere through long lead times, pushback from skeptics, and the usual challenges that come with product development.

Getting carried away with ML is easy: it's complex, challenging, and exhilarating. When you're starting out as a PM in this space, it's definitely worth investing time to understand how things work and experience the thrill of testing and validating your ideas. This expertise is crucial for building trust with your engineering team and earning credibility with your stakeholders.

However, over time, you'll realize that other skills—developing relationships with stakeholders, creating a robust strategy, and knowing how to pitch your product and team—are just as important as having a deep understanding of ML. Every professional sooner or later discovers that hard skills can only take you so far—unless you're truly exceptional—and this is equally true for PMs working in the ML space. As you progress in your role, it's important to stay up-to-date with industry trends, but don't focus too much on that knowledge alone. Developing soft skills is just as vital, if not more so, for the success of your product, your team, and yourself.

In my experience, people tend to either believe in ML or not, and this belief is deeply ingrained. From what I've observed, there are three archetypes: ML enthusiasts who are excited about ML without understanding much about the technology behind it, ML realists who work with ML and understand both its capabilities and limitations, and ML skeptics who are generally distrustful of the technology. If you find yourself surrounded by skeptics, it can be discouraging, but dealing with ML enthusiasts comes with its own set of challenges! Regardless of whom you work with, our role as PMs is to ensure that the right product decisions are made and supported by key stakeholders. So, be prepared to educate and align to achieve that goal.

Education. Your stakeholders need to have a realistic view of how ML works, how it is developed, what it can and cannot do, and how much it costs to be able to make informed decisions. 

Why is it important? A typical cause of ML skepticism is a lack of understanding; an unexplainable black box that takes away control from humans is a hard sell. Demystifying ML is how you make skeptics more comfortable with the black box and more open to hearing about its benefits. Education is also essential for ML enthusiasts who may have an over-inflated view of ML capabilities and costs. It's important to balance their excitement with realistic expectations.   

Alignment. With your key stakeholders now educated on ML capabilities and limitations, build a structured framework for deciding on ML investments. Once your stakeholders agree on the criteria and evaluations, reaching alignment is fairly straightforward. The main goal of this exercise is to persuade the skeptics, but it may also come in handy when checking your and your team’s bias towards using ML: “if all you have is a hammer, everything looks like a nail”, after all. Having a measured look at the costs vs benefits might curb your own enthusiasm. 

Keeping the framework simple is key, and I suggest having just two criteria: cost and benefit (assuming the investment is suitable for ML in the first place). When determining the cost, factor in ML and DE expertise, computing resources and infrastructure—for the launch, maintenance, and updates—and the opportunity cost too. When considering benefits, think long term: ML's unique ability to adapt to changing customer behavior, scale without sacrificing quality, and support continuous systematic improvement takes time to manifest.

On top of that, building an ML muscle can give a business a sustainable and hard-to-replicate competitive advantage; that expertise can be applied to various problems and applications. So while thinking about the full cost of ownership, make sure you consider the full benefit too.

It is easy to get caught up in the details of ML execution: it offers a huge range of optimization choices and it is fun to run and analyze experiments, testing different approaches and hypotheses. Proper ML requires a systematic, rigorous approach, and if you are academically inclined, it is very alluring. However, our job as PMs is to make sure that the team works on the most impactful problems relevant to the end user and the business, and that the great (and expensive) models our team builds get deployed in an optimum manner. Let me unpack what I mean here.

Firstly, it is rarely the case that the model’s output is directly presented to the user. In most cases, the model is part of a bigger customer-facing product: e.g. a targeting model decides to whom to send promotions but it does not generate the promotion content or presentation. However perfect a model might be, its impact is realized as part of a bigger product, and the entire bundle needs to work well. The advice is to diversify the integrations and work closely with your partner teams on making each integrated product a success. 

Then, there are often technical challenges with making sure the model outputs predictions as expected in the production environment and is retrained regularly to leverage new data on customers and products. For optimal performance, many pieces of infrastructure need to work reliably and in sync: model features need to be available and the scores need to be computed within the expected latency; retraining pipelines need to get updated training data and run reliably with a chosen frequency. As you can imagine, nothing ever works as planned. Extensive monitoring of key performance metrics and key processes together with a solid fallback is a solid approach to minimizing losses. Your infrastructure and data engineering teams are your best friends in realizing the full impact of ML models for the end users and for the business.

I hope you found the paragraphs above helpful. Did you have similar experiences with ML development? Is there anything else about ML that I didn't mention that you think is unique or typical? Let me know in the comments!