Experimenting with a Hackathon for PM’s

The results from Headlight‘s attempt to uncover New York’s most brilliant product managers

_Sign up as a competitor or employer for_👩🏾‍💻NYC Engineering Tournament👨🏻‍💻 (Apr 20–23)

Product management is one of the most coveted and difficult to land roles in tech. It is a multidisciplinary role which requires technical know-how, product insight, organizational chops, and people skills—and can bring huge success to a company in certain hands, and major harm in the wrong ones.

For candidates, it can be hard to definitively prove your ability to do the job.

For employers, it can be paralyzing to hire for a role that’s so important yet so hard to define, measure, or articulate.

This means most product management offers go to people who already have the the title “Product Manager” on their resume, with experience at a well-known tech firm or educational institution, or to an internal hire (usually an engineer or designer) with the hope of coaching them on the skills they’re missing.

This makes it almost impossible for people to break into the field.

I faced these challenges myself in breaking into product (you can read the whole story in FastCompany) and that’s part of what inspired me to co-found Headlight, a performance hiring platform. We help employers recruit, screen, and hire engineers, product managers, marketers, and other roles.

Source: The State of Tech Hiring 2017

Helping PM’s Get Hired for Ability, Not Pedigree

One of our company’s key tenets is that we should hire for what people can actually do, and not get distracted by the shiny things they have or don’t have on their resume. The truth is, decades of research has shown that work samples are better predictors of success on the job than years of experience or education.

Our own research on tech hiring has found that rather than people / culture skills, the biggest reasons most tech candidates don’t get jobs is lack of (or perceived lack of) technical skills. While take-home assignments are gaining popularity in screening developers, PM candidates haven’t had as many opportunities to demonstrate their ability to succeed in the role.

In December 2017, we released 9 PM take-homes across B2B, Consumer, and E-Commerce business scenarios.

At the end of 2017, Headlight launched PM Take-Homes, a free library of assessments / mini-projects that employers could use to screen PM candidates. We got some great feedback from folks who tried out these assignments but what we found was that they were still heavily relying on resumes and thus cutting down their candidate pool way too early.

I had run a one-day product workshop a few months earlier and found an enormous hunger within the NYC tech community for more opportunities to develop their skills and get recognized for their work.

So we decided to take things further and designed perhaps what was the first ever online product manager tournament.

The NYC Product Tournament

In January of 2018, we announced the NYC Product Tournament in partnership with Day100, a psychology-based hiring tech platform. The goal of the tournament was to help identify who were the top PM’s in the greater NYC area and help tech employers find their next product hire.

The Product Challenge

We wanted to create a product challenge that was interesting, meaty, and open-ended, to give people a flexibility to interpret and address the problem in their own way.

We had a little fun with it and turned the challenge into a bit of a Marvel fan-fiction.

Participants were put in the role of the newly hired Head of Product for Stark Labs — an incubator program inside the (fictional) Stark Industries.
One of the portfolio companies was struggling with the pilot of their FaceID competitor: SecureSmile.
The original pilot had SecureSmile put inside ATMs and consumers were asked to setup their smile data so they could get cash just by smiling
After the pilot saw limited adoption, the company is now wondering if it should iterate on the experience and try again or enter a new vertical like luxury apartment buildings, partnering with a smartphone manufacturer, or doing something else entirely

The challenge asked competitors to develop a product plan (in presentation format) that would be shown to Tony himself when he arrived in the office a few hours later.

They were given a template deck with about 15 different slide formats (big quote, list, 2x2 framework, etc) that they could use to get started. About half of finishers elected to use this and that’s why some of the decks have a similar style.

Here‘s a 5 minute video of the product challenge we presented to competitors. They had three hours to submit their presentation.

Click here to jump to the final presentations.

Prizes

To incentivize folks to participate, beyond just the glory of victory of course, we put together some prizes for top finishers. From a tech reading set of Reset by Ellen K. Pao, Hooked by Nir Eyal, and Sapiens by Yuval Noah Harari, to the latest innovation from one of the world’s most successful product organizations: Airpods, to a roundtrip flight anywhere inside the United States, we wanted to have this tournament to have some tangible stakes and outcomes.

Evaluation Criteria

We talked before about the challenge of figuring out who might succeed in the product role as being difficult because it requires so many different competencies.

For the sake of the tournament, we broke that down into four categories:

Business Opportunity: How well do they define a clear and compelling market and business case for this product? Do they appreciate the competitive landscape?
Customer Insight: How well do they understand customer needs, pain points and world view? How would they use research to inform this understanding?
Product Development: How well do they actually articulate a product solution? Do they understand the design and technology issues at play? Have they defined what success looks like?
Project Management: How strong is your execution plan for taking this product to launch and beyond? Are you able to organize the work in a practical and efficient way?

Psychology Assessment by Day100

Since we were experimenting with a new kind of hiring process where we hid candidate’s names and even resumes, we wanted to give employers a chance to learn more about candidates, so we turned to our friends at Day100, a psychology-based hiring technology platform that works off the Big Five personality framework that is most widely accepted by psychologists.

Each competitor completed a Day100 assessment, which was reviewed against an “ideal PM temperament” that we created at Headlight. This wasn’t perfect as every company and manager has a different idea of what they’re looking for, but it at least helped reveal more about what a candidate was like.

Judges

To bring it all together, we needed to have capable and respected folks reviewing the work of the participants. We assembled a panel of folks from four fields: product, engineering, marketing, and design. By having multiple judges review the work, we’d have a more balanced and healthy view of the participant’s ability.

We recruited judges from different size companies — startups, midsized tech co’s, and publicly traded firms — and in different industries — ecommerce, SaaS, media, developer tools, and social networking.

I have to give a huge shout out to Cliff Kim, Kate Zasada, Veronica Ray, Frantz Joseph, Jordan Obi, Joel Califa, Sam Yeh, and Victor Gamez for giving up their Sunday to churn through dozens of competitor submissions, providing feedback and debating the merits of top finishers to get to our final list. #YouTheRealMVP

The Competition

On January 4th, we announced the NYC Product Tournament. We shared the tournament with folks in our networks, on private Slack and Facebook groups, on social media, and through our email lists.

In just over two weeks, we had 153 PM’s register to compete!

Who were these folks?

Most of them were already product managers, with a majority of the remainder in a role adjacent to product like engineering or product marketing.

They were nearly evenly split between men and women and 75% of them were between 25 and 34 years old. About half were Asian-American, another 35% were White, and 16% were Black or Latinx.

Making tech a more inclusive place where people of all backgrounds can thrive is something we care a lot about and we’ll continue to work on making our tournaments accessible, especially to traditionally underrepresented groups.

One final stat — the majority of respondents were actively looking for a new job (60%) while the remainder were open to new opportunities. This isn’t far off from our survey findings which indicated that 75% of tech workers had either changed jobs or seriously considered a move in the past year.

Employers

We reached out to a couple dozen NYC employers with PM openings and about 14 folks said they were interested in seeing top candidates.

Final Results

Over the course of Jan 20–21, we had 53 people work on the Product Challenge. Each competitor was given only three (3) hours to review the prompt, develop a plan, and put a presentation together to convey their ideas.

Every submission was reviewed by at least two judges, and the top scoring presentations (normalized for tough vs easy graders) were identified.

Because we emphasized blind review, judges had nothing to base their judgement on except for the candidate’s work. We even code named these candidates with planets from Star Trek so their race and gender were hidden.

The top 14 of competitors were identified, which represented the top quartile of finishers.

We had all judges look through the top 14 finishers and together they all independently voted for their Grand and First Prize winners. So far, one of those PM’s have gone on to be hired by an employer via the Tournament.

Analyzing our Top Finishers

While I’d love to tell you about why all our top finishers (top 25% of all competitors!) are awesome, I’ll stick with just our top four for brevity’s sake.

🏅 First Prize: Benthos. This competitor proposed a new product line geared towards music festivals. Our judges felt like there was a bit of a leap there and wanted more details on the business model. That said, they laid out the feature set very clearly, with effort / impact estimations on the MVP. Their KPI’s for success were also well-considered and overall the concept was strong.
🏅 First Prize: Onara. This competitor proposed a new product line around luxury apartments. Judges felt they made a good case for the new market and appreciated the user personas (even if they were probably made up given the 3 hour timeline). The presentation also included good thinking on revenue models and used the presentation template well, though the project plan could have used more details.
🏅 First Prize: Finnea Prime — This competitor proposed a new product line around cruise ships. The presentation featured some solid research on how different organizations are using facial recognition and included a 2x2 matrix analysis of different market opportunities. That said, the actual cruise ship proposal, though creative, did not get adequate justification and the product plan was light.
🏆 Grand Prize: Talax. Our overall winner chose to stick with the banking market. They presented thoughtful ideas for why the initial pilot failed and had solid thinking around customer segmentation and a clear product concept. They were also one of the few who identified risks with the project and how to mitigate them. The only real ding on this one was that the deck was extremely text-heavy with very little visual presentation — but since the focus was on the strength of their ideas, this didn’t hurt Talax much in the end.

All Presentations

Below you can see an Airtable of all our top finishers. Interested in meeting any of them? Shoot us a note at team@headlightlabs.com with your company and a relevant job description and we’ll be in touch!

Lessons Learned

We started this journey wondering if it would be possible to run a hackthon-esque competition for product managers and I think the answer: definitely!

Overall our competitors found the product prompt to be clear, appropriately challenging and interesting. That said, there were definitely still areas we could improve:

Give people more time / flexibility. We wanted to give the challenge the same kind of time pressure that a hackathon could have. Three hours is clearly a tight time-frame and I think that stressed people out. We also could have extended the timeframe in which you were able to compete from just the weekend. I wrote a note to many of the people who never submitted anything and the vast majority of responses indicated that an unexpected event came up that weekend that prevented them from competing. Our Engineering Tournament extends the timeframe to 4 hours and doubles the number of days it’s open (Friday — Monday).
Give judges more context. We tried to give our judges a sense of the range of submissions and thought our rubric / evaluation criteria was clear, but once they actually started grading, they found it hard to determine actual scores. Sometimes a presentation’s business thinking would have been a 3 stars for a senior PM but 5 stars for a junior PM.
People crave feedback. As a two-person team, we were trying to do a lot, between introducing top finishers with employers, supporting our users on the Headlight platform, and talking with investors. That said, I was struck by how many competitors kept asking when they were going to feedback from judges (which we had to organize and collate) and also how many people opened and clicked through to the presentations of our top finishers. It’s clear that learning how they could improve (as well as studying what top finishers did) was a key reason for why many people entered.
Employers need convenience and context. Due to a poor employer UX (literally a spreadsheet with links) and not enough context on our candidates, only 5 of the employers actually took the time to review all the candidates and request introductions. That’s why I made the Airtable to better showcase the work. We’ll also be asking top finishers to write a small personal statement about what their career goals are to help employers understand what they’re interested in / looking for in their next role.

All in all, the Product Tournament was a lot of work, but I’m really happy we were able to pull it off. Probably the best feedback is that 81% of competitors would recommend something like this to their colleagues / peers.

We’re excited to take what we’ve learned from this Tournament and apply it to our next event — the NYC Engineering Tournament 2018.

If you’re interested in having better onsite interviews and reducing time spent in the screening process, check out Headlight. And please give this piece 👏 so others can find it too!

Have thoughts, questions, feedback? Find me at @jasonshen.