Problems in Feedback Collection

A major part of a product manager’s job is to setup experiments to understand the user better. In order to do this, a smart product manager should learn to get feedback from his users – sometimes overtly, sometimes covertly. There are several ways this can be done, and an earlier post outlines some techniques using which this can be done. This post talks about some of the problems in feedback collection.

1. Spotting difference between what people want & what they say they want

The first, and perhaps the most common problem is understanding that there is a difference between what users say they want and what they actually want. There is a famous quote by Henry Ford (although not said by Henry Ford actually), that he asked people what they wanted, they would have said that they wanted a 1000 horses and not an engine powered car!

galloping-horses-745828  fast-car-models-cars-pictures-and_29852

The key here is in noting the difference between wanting a 1000 horse powered chariot and something that just travels faster. It is important & event mission critical to understand this difference in user need. Thankfully, Henry Ford never asked his users and simply built an automobile vehicle. The rest, as they say, is history.

2. Wording of Question

Another common problem in feedback collection is the verbiage – the choice of the words used in questioning. Asking a leading question is not the way to go (unless there is obvious reason to do so). Asking an open ended question is much better way to go about things. I’ll give an example.


Asking  ‘Do you find this product/service useful?’ is an example of a leading question. This leads the respondent to say even though he may not mean it (may be the respondent says it to not sound harsh). Instead asking ‘How/What do you think/feel about this?’ is a much better question to ask.

3. Selection Bias

When you decide to talk to users, it is plain and simple right? Just pick some random guy up and get him a coffee or doughnut and spend the next 15 mins asking him basic questions. The process sounds simple, but as always the devil is in the detail.

When selecting people to do an interview or a face-to-face discussion people tend to pick people who are most likely to confirm our believes than people who are likely to say a different thing. This phenomenon is known as selection bias.

So do a study to categorize people into different groups. Then, expect them to behave in a certain way. Frame these expectations based on your gut and past experiences. Now as you interact with different groups of people, readjust your beliefs about that group.

4. Approximation Problems

Segmenting user groups may be an easy solution on paper, but is hard to implement in reality. For example, who would one identify the segments in the users of a mobile chat app. What is the best way to segment users – according to age, gender, primary purpose of usage or a combination of the above?

This is a question that has no clear and direct answer. The question is that it depends on the context of the analysis. If the decision is on the color of the app design, then segmenting by gender sounds like a good way. If it is for font size, then segmenting by age group sounds better. There is no obvious answer, but an approximation is necessary to proceed with the decision making. Be careful to realize the assumptions upon which you are basing your decisions.