Typical user survey at tech companies
“The story is told of a focus group for a new $100 electronic gadget. The response in the focus group was fabulous. All the people talked about the features of the new device with excitement.
At the end of the session, the moderator said “Thanks for coming. As our gift to you, you can have your choice of the device or $25.”
Everyone took the cash.
Surveys that ask your customers about their preferences, their net promoter intent, their media habits — they’re essentially useless compared to watching what people actually do when they have a chance.”
— Edited excerpt from Surveys and focus groups by Seth Godin
Seth Godin hits the nail on its head with this one. Surveys that ask users to predict their own future behaviour or their opinions are absolutely futile. Humans are famously inept at predicting their own future actions, not to mention the numerous biases that creep in when users are asked for their opinions. Yet most user surveys today do precisely that.
But that doesn’t mean surveys are obsolete. You don’t always have the luxury of observing all user behaviour, and sometimes surveys are necessary.
How do you then run a survey to avoid these pitfalls?
The golden rule of user feedback:
Get facts, not opinions.
Ask questions that get your customers to share facts and experiences rather than opinions. Don’t ask “Would you..?”. Ask “When is the last time you..?” or “Tell me about a time when you..?”.
When you’re trying to gauge for a potential product or feature idea, instead of, “Would you use this?”, ask, “Which features do you use a lot?” and “When you use feature ‘x’ a lot, what are you specifically trying to accomplish?”
Talking to users is hard work, but it’s a price you’ve got to pay on the road to building a great product.