Features Add Ons, Product Management Techniques

Collecting feedback and associated trade-offs

This was my response to a question in one of the Slack Forums I’m a part of. I wanted to share it here because I think it would help PMs faced with a similar (seemingly trivial) situation.

Question:

In your experience, how valuable are in-product surveys (like asking “How did you like your experience? :thumbsup::thinking_face::thumbsdown: ” during the user flow, or first conversion, etc)

Answer:

It depends. From my experience, this has not been valuable. I’m in the camp of asking for feedback just once per transaction (assuming yours is an e-com product). And post-order fulfillment NPS is a well understood standard for this. I do understand the temptation for an individual team to do this as a more direct way to measure their part of the experience.I’ll tell you the history of how this evolved in my org and why we didn’t do this.

I was responsible for NPS at my last org and we were a Naspers portfolio company. We are an online travel aggregator. The NPS metric would be reported all the way up to Bob (CEO of Naspers). The leadership team at Naspers would compare NPS across their portfolio of companies and ask founders/CEOs on trends. Given the importance, there was an external audit team (likes of KPMG) who would come in every year and check if we are counting repeat users, repeat clicks, inclusion of cancelled orders, inclusion of payment failures, how many times we would ask users and on what channels – pretty elaborate. They would give us a list of changes to make this a fair comparison across the Naspers portfolio.

Around 2018, management started linking NPS targets to variable payouts – in a bid to improve NPS. This set of a race within teams to address questions about the quality of the part of the experience they owned. Ex – the transactions team started doing what you mentioned, asking for a simple feedback after a conversion. Since they were a strong PM/tech team, they pushed this task in one of their sprints and even started reporting this. Their argument against NPS was that it was a motherhood metric. It was good as an output metric but not useful as an input metric if you wanted to focus on improving the experience. This was a fair argument. Multiple other teams too wanted to do something like this. However, we started fearing for the end user’s experience if they received multiple feedback prompts – one for booking, one for order fulfillment and for other touch points. As you can see the conversation quickly devolved into an undesirable dynamic. We either had to arbitrate which teams had the special right to send feedback prompt or dismiss the validity of their arguments. Both seemed like bad ideas.

My side of things – this development made my job harder because I was being asked questions on NPS, but the teams I depended on (each) wanted to use a proxy. Sure, there would be some correlation between the metrics. But I would have to track each one, ensure the same rules applied in collecting the metric and manage the narrative on how they correlated. My team was too small to do all of this, we also had other metrics to focus on. I was trapped in a problem that a lot of horizontal product teams face.

Luckily for me two things happened. First, the response rate of feedback prompt post conversion was low. We learnt that after a conversion event users tend to be distracted. They are busy double-checking if they ordered the right thing, right quantity and other details. They were also checking the communication from their payment provider, if the right amount was deducted. Second, management insisted that they were interested in NPS and not an intermediate metric. Teams could use separate metrics, but team leaders were measured by NPS and nothing else. So if they were to use a separate metric then they had to answer how this correlated with NPS. Also, management delegated the concern of the negative experience of multiple prompts to the relevant product folks (me) to decide on. This clarity greatly helped me.We eventually settled on using only the NPS. Each category/subcategory within NPS would be used as an input metric for individual teams. We avoided using a post conversion feedback prompt because the incentives were set that way.

I later learnt that NPS was intentionally kept as every team’s shared metric. This reflected the fact that quality of user experience is, at the end of the day, every team’s concern – even if the dots to linked incentives didn’t seem obvious.

All of this took 6 months to play out. And yes, we did improve the NPS in the final quarter that year.

I’m not sure if this helped your question. Hopefully it gives a broader perspective to make a decision for your context.

Standard
Uncategorized

imparting urgency

miten sampat blog

August 2020

one of the tougher challenges in being an entrepreneur/manager is imparting an appropriate sense of urgency to colleagues & collaborators.

what is urgent, what is important, and what’s the right sequence of priorities at any given point changes from context to context (for individuals, teams, counterparts). Yes, there are umpteen tools & frameworks to enable mapping out tasks, but at many levels these choices are filled with ambiguity.

great teams get this right, andit’s not just the responsibility of the leader to make these trade-offs; done right this is a collective process that is driven by external inputs as much as internal goals & desires.

there are no right answers, but here are some of my thoughts on what could work:

  1. Cadence of communication — teams that communicate with a predictable cadence end up having a greater sense of “what and why” to build/execute towards.
  2. Articulation of first…

View original post 230 more words

Standard
Uncategorized

Getting back

I’ve not been active on this blog for several years. I first started it as a way to talk about Product Management and what was in my head when I was starting out as a PM.

Several years down the road, I feel my thinking on these topics have evolved. I also feel like I can read the tea leaves better. So, with a renewed gusto, I feel an urge to restart this. I have more to say and with better clarity. I hope you find it worth your time.

Here goes my renewed attempt at keeping this space active.

HD wallpaper: photo of person typing on MacBook Pro, laptop, hand ...
Starting this again
Standard
General, Product Management Techniques

Problems in Feedback Collection

A major part of a product manager’s job is to setup experiments to understand the user better. In order to do this, a smart product manager should learn to get feedback from his users – sometimes overtly, sometimes covertly. There are several ways this can be done, and an earlier post outlines some techniques using which this can be done. This post talks about some of the problems in feedback collection.

1. Spotting difference between what people want & what they say they want

The first, and perhaps the most common problem is understanding that there is a difference between what users say they want and what they actually want. There is a famous quote by Henry Ford (although not said by Henry Ford actually), that he asked people what they wanted, they would have said that they wanted a 1000 horses and not an engine powered car!

galloping-horses-745828  fast-car-models-cars-pictures-and_29852

The key here is in noting the difference between wanting a 1000 horse powered chariot and something that just travels faster. It is important & event mission critical to understand this difference in user need. Thankfully, Henry Ford never asked his users and simply built an automobile vehicle. The rest, as they say, is history.

2. Wording of Question

Another common problem in feedback collection is the verbiage – the choice of the words used in questioning. Asking a leading question is not the way to go (unless there is obvious reason to do so). Asking an open ended question is much better way to go about things. I’ll give an example.

questions

Asking  ‘Do you find this product/service useful?’ is an example of a leading question. This leads the respondent to say even though he may not mean it (may be the respondent says it to not sound harsh). Instead asking ‘How/What do you think/feel about this?’ is a much better question to ask.

3. Selection Bias

When you decide to talk to users, it is plain and simple right? Just pick some random guy up and get him a coffee or doughnut and spend the next 15 mins asking him basic questions. The process sounds simple, but as always the devil is in the detail.

When selecting people to do an interview or a face-to-face discussion people tend to pick people who are most likely to confirm our believes than people who are likely to say a different thing. This phenomenon is known as selection bias.

So do a study to categorize people into different groups. Then, expect them to behave in a certain way. Frame these expectations based on your gut and past experiences. Now as you interact with different groups of people, readjust your beliefs about that group.

4. Approximation Problems

Segmenting user groups may be an easy solution on paper, but is hard to implement in reality. For example, who would one identify the segments in the users of a mobile chat app. What is the best way to segment users – according to age, gender, primary purpose of usage or a combination of the above?

This is a question that has no clear and direct answer. The question is that it depends on the context of the analysis. If the decision is on the color of the app design, then segmenting by gender sounds like a good way. If it is for font size, then segmenting by age group sounds better. There is no obvious answer, but an approximation is necessary to proceed with the decision making. Be careful to realize the assumptions upon which you are basing your decisions.

Standard
Uncategorized

Product Culture in Top Orgnizations

product ninja

To an outsider, the culture inside the hottest tech companies in the world might sound the same. They all have free food, flexi hours, no dress code and have a ton of perks. But when someone talks about culture and mentions only these things, one can be sure that this person has no idea what work culture actually means.

Culture has a lot to do with how things get done. It deals with the power centers in decision making. It has a lot to do with the level of influence each team has in shaping the final product. Product management is certainly an important function and for a young product manager it is important to know the level of influence the Product Management team has in a company he wants to join.

Read this question on Quora. A famous product manager shares her view on the most influential team in various hot tech companies. She goes on to say that in Google & Facebook it is engineering team that calls the shots. My encounters with an engineer at Google also suggests the same. (Although I admit that judging based on a one person sample is criminal).

Reading the post, Microsoft, Yahoo and Amazon have product managers who are real ninjas. They seem to hold the maximum influence in decision making. This is understandable too. Amazon & Yahoo have the best internet strategies. Although Yahoo’s is a tad too unique for other companies to follow. But this is strategy and I guess it should be unique!

And finally. Surprise, surprise! In Apple it is a shared spot at the top between the Executives and Designers.

Standard
Uncategorized

The Biggest Mistake a Product Manager can Make is a Big One

dilbert

A product manager’s role is to make successful bets on the product and take the product to a higher orbit. But making a bet also implies that some decisions will be successful and some will fail. For a young product manager this situation presents a dilemma. Does he back himself and take a risk. Or does he simply follow orders and remain the guy who communicates a vision and not stick his neck out? Obviously, if the opportunity presents itself, a product manager should propose a feature and make the bet. But how does he ensure that the downside of this bet doesn’t kill him? Based on the conversations I’ve had with product managers, the top 1% product managers try something like this.

Continue reading

Standard
General, Industry News

Living Life in Beta

This philosophy is truly timeless. It was advocated in ancient Hindu mythology and then in the teachings of Buddhism. In more recent times, it was known as the Kaizen principle. All of these point to one thing. Keep improving. Period.

Any product manager will tell you that the beta phase is a hard one to navigate. The product has some obvious bugs, doesn’t deliver on promises and creates self-doubt in the minds of those who once believed in the vision. It’s tough phase to live in and requires a lot of skill and experience to navigate through it. The top 1% product managers know a way to power through this. And they stress on continuous improvement.

Continue reading

Standard