A survey for steering
A quantitative study on user needs and possible solutions
Once upon a time...
there was a trade ship stranded on a mystical island in unknown territory. The seafarers could not come to an agreement on how to best market their goods, while steering clear of dangerous waters.
They thus assigned the ship's navigator with the task to map the surrounding islands and help them find a clearer path.
Project steps
1. Kicking-off & recruiting
"Are our ideas for future product development meeting the target group needs?"
A screenshot of the sample distribution in the survey, representing the target group.
-
What happened?
In a series of strategy off-sites, the management aimed to outline the future course of product development. They asked the UX team to verify their assumptions of user needs and to help prioritize solutions.
​
-
Which method to use?
Given the large scope of the study and impact of the results, I
suggested to conduct an online survey with a representative sample.
Challenge: The request came short-termed and with limited resources. Although we were not able to explore new user needs, this closed method allowed us to reach a two-week deadline with a large sample size (n=850).
-
Who are our users?
To ensure participants represented the target group, I conducted desk research on previous market segment studies and inquired with internal experts.
2. Constructing the questionnaire
From listing features to identifying needs
A screenshot of the miro board provided by the management with their collected ideas for future strategy focus areas - the basis for constructing the questionnaire (anonymised).
​Challenge: The questions from management were solution-oriented, focussing on features instead of user needs.
​
-
What happens when users are asked if they like a list of specific solutions in a survey?
-
A specific solution is difficult to envision for participants.
-
Typically, users respond positively to offered solutions, which does not predict well if and how they will use it.
-
Results won’t show if a different solution might also work or would even be preferred.
​
Instead, we need to verify underlying needs. Then we can be fairly confident that a solution targeting those needs will be valued by users.
Solution
I proposed splitting the survey into two parts:
-
Part A: rephrasing the 7 suggested solutions into 7 needs to represent the underlying user needs
-
Part B: asking users to rate the importance of the 7
suggested solutions, to cross-check the answers with the results of part A
Creating tangible items and defining the answer format
An example question for part A aiming to explore one out of seven proposed user needs with 5 items (partly inverted, randomized) on a 6-point agreement answer scale.
-
Why 5 items per user need?
To ensure measurements of a user need were reliable and valid. In other words, we covered the need well enough because a portion of items may be misunderstood.
-
Why positively & negatively phrased items?
To prevent fatigue of participants and to identify participants that showed speeding or patterns during the data quality check.
An example question for part B aiming to prioritize one out of seven proposed solutions with 3 items (randomized) on a 6-point importance answer scale.
-
Why fewer items - actually one per sub-feature? Solutions (and features) were much more explicit and easier to understand for participants than needs. Fewer items were sufficient.
-
Challenge: With now in total 56 questions the survey was too long. We risked that participants reply with less care, possibly leading to wrong results. We thus asked each participant a subset of 4 out of 7 randomly selected needs and solutions.
Once upon a time...
...when all the questions were gathered, the ship's navigator ventured out to find answers.
She flew as high as she could to see far and wide; sent out birds to venture even further and report back; she used her navigation instruments to measure, gauge, meter and calibrate with all her skill.
She went above and beyond, determined to help her crew to find the best route ahead.
Conducting the survey in Alchemer
Screenshots of Alchemer (top: questionnaire building, bottom: quota overview), the tool used for conducting the online survey.
​1. Preparation phase: set up of...
-
legal information
-
screener questions for logic based quotas
-
randomization of questions & items
-
redirect links with recruiting agency​​
​
2. Conduction phase:
-
soft-launch of 10% after rigorous internal testing
-
full launch after soft-launch review
-
monitoring of quotas & communication with recruiting agency
​
3. Completion phase - data quality control:
-
excluding non-complete data sets
-
excluding duration outliers
-
excluding straight-lining participants & others with incoherent answer patterns
​
​
What does the data tell us?
A screenshot of my analysis board.
​The two key questions we wanted to have answered were:
​
-
Which user needs are the most important to be addressed first?
To answer this question, I ranked the proposed user needs of part A by their average mode (on the agreement response scale).
-
Which solution ideas are the most promising ones to pursue?
To answer this question, I ranked the proposed solutions of part B by their "top 2 agreement sum" (in %).​
​​
Challenge: In one case a user need that was expected to rank very high, contrarily ranked very low, meaning it seemed like it was rather unimportant. However, the solution that was proposed aiming to address this need was ranked of highest importance by users in part B (solution). I critically reviewed the items measuring the need and realised that their phrasing might have skewed the outcome. I advised that further testing would be required before drawing final conclusions.
​
​
4. Sharing the results
_edited.jpg)
I presented the results to the management in 20 minutes presentation, highly condensed to the essential key findings and packed with actionable insights.
​
Outcome:
-
The assumed user needs had mostly been confirmed.
-
Thus the also proposed solutions could be assumed to address actual, underlying user needs.
-
The ranking of user needs by the importance of their proposed
solution, was received by management as a valuable input to define their strategy. -
The findings helped to settle a few ongoing disputes about the relevance of developing certain features.
​
Now, the management knew that they were heading in the right direction - and could decide which need to tackle first to have the most business impact!
Happy End!
Once upon a time...
...at her return, the navigator retreated to her tent, not allowing any visitors to disturb her brooding over the things she had learned.
One day, however, she stormed out beaming, wildly waving a carefully drawn map. She lay it out in the middle of her assembled crew and explained everything in detail.
​
Equipped with such map, the traders could finally settle their dispute on the best route - and went on shipping their goods to island creatures and evading all dangers, without ever going astray.
And so they all lived happily ever after.
