Sometime back I had tweeted a thread about how I dislike NPS score as a measure of success. It led to a fair amount of discussion and debate. Hence, I decided to dig a bit deeper and write a longish post about it. While I have tried giving it some structure, each section is more or less self contained. You can directly jump into a specific section if you are already aware . I have tried to give enough context wherever possible.
I have also linked to sources wherever applicable, so feel free to follow them and do your own due diligence when in doubt.
Disclaimer: I am not yet fully satisfied with the article and feel I add more evidence, but this has been in my inbox for ages. I may add more details and address any specific questions and criticisms that may come my way. Please do let me know if I misrepresented or missed something
What is Net Promoter Score AKA NPS
Net promoter score is a widely used measure of customer loyalty today. It’s claim to fame is its utter simplicity. It can measure customer loyalty with just 1 question.
To calculate Net Promoter score you ask a statistically significant number of your customers to answer a single question
How likely is it that you would recommend [brand or company X] to a friend or colleague?
- Ask them to select from a scale of 1-10(some orgs use a slightly different scale but 1-10 is the most widely used), where 1 means not likely at all, 10 means very likely and 5 means neutral.
- Anyone who choses 1-6 is considered a Detractor
- Anyone who choses 9-10 is a Promoter
- Net Promoter Score= %age of Promoters – %age of detractors
The basic claim of NPS is that it can reliably measure customer loyalty and if the company focuses its efforts to increase NPS, it can lead to more healthy growth.
This two specific claims are important to keep in mind
1) Reliably measure loyalty (better than other scores)
2) Correlated to company growth (see more details in NPS Origin story sec)
NPS origin story
While internet is filled with how to use NPS, when to use NPS, and why to use NPS, before we get to all those questions it is necessary to understand how NPS even came into picture.
The origin story of anything reveals a lot about the motivations without the burden of muddled history between then and now.
NPS score was invented by Fred Reichheld who was a consultant at Bain and company . It was introduced to the world in this HBR article
The basic idea came about when they looked at a car company use a very simple method to increase customer loyalty. The company, Enterprise Rent-A-Car, simply asked people two questions
– Quality of rental experience
– Likelihood you would rent again
The company then counted only those customers who gave it the highest scores on both the questions. All their outlets were then asked to optimize for this specific score. It was believed that this would inspire the sales agents to be better and increase customer loyalty.
Fred wanted to make this system much more simpler and see if this could be brought down to just 1 question
Process of finding the One Question
- 20 questions were created on the Loyalty Acid test survey
- Test was administered to customers in following industries
- Cable and telephony
- Personal Computers
- auto insurance
- Internet service providers
- Then they asked each participant to tell about a specific instance when they actually referred the company to someone. If this was not available they waited 6-12 months and asked again. This data of about 4000 users was enough to create 14 case studies which established a link between survey response and actual referral
- The top-ranking question was far and away the most effective across industries:
- How likely is it that you would recommend [company X] to a friend or colleague?
- Two questions were effective predictors in certain industries:
- How strongly do you agree that [company X] deserves your loyalty?
- How likely is it that you will continue to purchase products/services from [company X]?
- Other questions, while useful in a particular industry, had little general applicability:
- How strongly do you agree that [company X] sets the standard for excellence in its industry?
- How strongly do you agree that [company X] makes it easy for you to do business with it?
- If you were selecting a similar provider for the first time, how likely is it that you would you choose [company X]?
- How strongly do you agree that [company X] creates innovative solutions that make your life easier?
- How satisfied are you with [company X’s] overall performance?
- The top-ranking question was far and away the most effective across industries:
Link between NPS score and company growth
Then they tried to find correlation of NPS score of customers with the actual company growth
- In airlines a strong correlation existed between “Would recommend” question and average company growth
- Similar results existed in rental car business
- “Would recommend” was irrelevant for database software or computer systems as people had limited choice, and senior execs who made the choice were not part of the people surveyed. For such industries “Sets standards of excellence” and “deserves your loyalty” were far more predictive
- NPS was also not a predictor for Local telephone and cable TV company growth because they were near monopolies. Their growth was determined by how fast the population in their area increased
Who uses NPS today
Pretty much every one. As of 2020 2/3 of fortune 1000 companies seem to use a version of NPS. One simple experiment would be to search for the term “How likely” in your Inbox
Good things about NPS
It is very simple to measure and benchmark .
- Its a single question and is used by multiple industry players to benchmark against competition and internally
- Its easier to digest at almost all levels of abstraction
High completion rate
With users being inundated with all kinds of brands seeking their attention, it is much more easier to get them to answer 1 question rather than multiple. Infact in the paper “Assessing treatment outcomes using a single question” where they did an NPS of patients, they found that the NPS question consistently had the highest completion rate (96.5%). I would also now assume that it has become so common that users almost expect this question and are willing to answer
It defines loyalty in interesting way:
While loyalty may traditionally be defined by retention, LTV, and other metrics , it can miss out on word of mouth. NPS attempts to target that specifically by a bit lose but interesting way
Customer Loyalty Definition in Original NPS system
- Customer loyalty can be defined as customers willingness to stick to certain provider even if they are not providing the best possible rate in a particular transaction. Think if this like : ” Sure you may be charging me more today, but I know you have done great work in the past and generally give me good rate so I will stick to you even though cheaper options may be available”
- Customer loyalty is also more than just retention because some people maybe retained just because they cannot move out due to inertia, or exit barriers. Eg : Monopoly players , or prepaid plans
- Loyal customers may also not be repeat purchasers, eg when they outgrew that service. Eg: You may no longer buy a pulsar bike because you no longer drive a bike, but you would recommend it to your nephew when he is considering one.
Issues with Net Promoter Score
Its unclear what does it even link to
There are multiple leaps of faith on the original research. The way I read the original article it says
- Answer to NPS question seems to be the highest correlated among other questions to actual referrals in some industries
- Higher NPS seems to be correlated to higher growth rate irrespective of company size
It is unclear if this is even a measure of loyalty at all. It is simply supposed to be used as a predictor of growth
If you want to use it as a predictor of growth, the way it was actually intended to be used, it is unclear that it is in anyway better than say CSAT or good old churn rates / retention rates
Its arbitrary and ignores all cultural nuances
There seems to be no clarity on why someone who says 6 vs 7 are in a different bucket while 5 vs 6 are not. Also no clarity why focussing only on difference between promoters and detractors matter. What if we just tried increasing average score?
It also ignores all cultural nuances. Eg: if you travel in Uber in US vs India, you may see a huge difference in your ratings. Anecdotally I have seen my ratings drop in india and rise in US. I presume there is a cultural difference here. In India low rating is 1* whereas in US its 4* .
It is not reproducible
This is perhaps the BIGGEST issue with Net Promoter score. The biggest claim with NPS was it is the single best predictor of growth, but this 2007 paper found no support for that claim when they tried to replicate the same study that Reichheld did. Not surprisingly, they found that NPS performed as well or as poorly as using the customer satisfaction index to predict growth
There seems to be no real statistical backing to NPS, and as per the paper, even Reichheld acknowledges that
It is more noise than signal
What do you do with NPS? One common theme is that you work towards increasing it, but that is not a good reason to ask this question in the first place. What if you simply relied on satisfaction surveys or measurable things such as actual referrals , cost of acquisition, churn rates etc. There seems to be no clarity that it gives you anything additional
Another issue is that it varies a lot on multiple factors.
More variables = more errors.
Eg: when NHS introduced NPS they found that only about 40% variations in NPS scores was explained by overall satisfaction whereas rest was explained by various other metrics such as: if the patient undertook hip replacement or knee replacement.
Even the NPS difference between patients who underwent Hip replacement(71) and knee replacement(49) were stark making it impossible to benchmark them. A bad action item for the hospital would be to target for same NPS across all services.
If a hospital cannot even benchmark within its own departments, it’s useless to try and benchmark to others.
Now imagine Flipkart doing NPS. It would vary widely based on specific products a user buys. . This is something that is already captured in the product reviews, but NPS may unnecessarily create noise. Trying to make sense of these variations maybe leading to large time sinks
Some companies, especially in ecommerce, go overboard and try to find NPS for each product and service, which again seems unnecessary. It is no longer a measure of loyalty but just a feedback of the product, which may be better asked directly via ratings and reviews.
Eg see below Myntra trying to do NPS linked to a specific product. But even if I rate it low they may have no action item because they do not control the product itself. Lot many questions need to be asked to even understand my response.
A better question would have just been do you like the product, which not only would be direct but also feed their rating system.
Unclear if its worth spending effort on
Even if we assume that increasing it would lead to growth, it’s anyone’s guess that how much that number may be. If say I have an NPS of 70 and my competitor has 35, am I growing twice as fast?
It is as easily manipulated as any other survey measure
Just like satisfaction surveys which can be manipulated, so can NPS. Simple techniques could be
- Asking the user to rate you after a good interaction. Eg: As soon as order is delivered, or a ticket resolved. At this time you are no longer trying to find and fix issues, you are simply trying to get that score. This technique may have an effect for Play store reviews and youtube videos where ratings and likes are a social signal to other users, they are counter productive for NPS , unless this NPS is being collected as a vanity metric. Eg: Pitch deck, Presentation to leadership
- Incentivising the user: Eg give your software product for a free trial and see almost every customer give you high ratings on NPS. It is meaningless and may have no correlation to your growth
I also did a very unscientific survey on Twitter and Linkedin to know if companies took NPS targets, and if the person responsible for the target also controlled stuff like when NPS was sent and how to pacify the user: Here are the results
As you can see ~30% companies have NPS goals and the owner of the goal optimises of things like when to send the survey and in some cases even customer incentive . You get what you optimise for and in this case my hypothesis is that system is designed to make the NPS go up not necessarily the loyalty
You get what you optimise for
This maybe the reason why companies with really high NPS also go bankrupt
Its unnecessarily complicated
There are many reasons you would not recommend a specific product to someone. Just like in NPS article they mentioned that a loyal customer may not be a repeat customer because they outgrew the product, but would happily recommend it to someone who did not. Using the same logic, someone who may be a loyal customer may not recommend it to friends who may not be the target customers. NPS would classify these loyal customers as detractors.
Another issue is that it assumes that detractors are not promoters. But the reality is different. Your personally may hate the product but if you were to suggest a product to someone you would play a matchmaker role and take into consideration their individual needs and circumstances.
This 2019 survey found that 52% users who actively discourages others from using a brand also actively recommended it. You can be a promoter and detractor at the same time based on who you are talking to or how your last experience with the brand has been.
It’s not a single question
While the whole USP is its single question, you invariable would need more information as soon as the users rate <7 , defeating the whole purpose of simplicity.
Loyalty is multidimensional
While NPS seems to acknowledge that loyalty is multi dimensional, it tries to collapse it into a single dimension of word of mouth.
The purest measure of loyalty in my opinion is customers actually spending money to buy your product. I criticise my bank a lot, but despite many many alternatives I have stuck with them for 15 yrs. By every definition, I am a loyal customer who they would want. A research done by University of Cambridge with an asset heavy company in UK over three years they found that NPS and actual user behaviour did not match
Its probably not for your industry
This is less of NPS issue and more of marketers abusing NPS because of its perceived simplicity. Even in the original research NPS was not found to be a predictor of growth in industries such as computer databases. Remember the ONLY thing it was supposed to do was predict if you will grow, without that correlation the score is more or less useless. Sales is complex and any industry with high inertia, top down decision making, and monopolistic players NPS is not even applicable. This makes me wonder why so many startups are obsessed with it.
Its also possible that NPS should not even be a big goal. Eg in the NHS paper I referred to, difference in NPS among patients was was not due to actual patient care and recovery. Perhaps NPS is not even a measure for hospitals.
NPS seems to be a arbitrary score with little statistical backing. It may not even be valid for many industries which are currently using it.
While it can be used as a tool in your armour of many other signals, over reliance on this for making decisions is not prudent.
NPS is popular perhaps because it is simple, but this reminds me of the phenomenon of Bikeshedding .
Bikeshedding: If a committee were to design a nuclear power plant, they may spend far more time than necessary to discuss the bike sheds, its color, its position, and its capacity . The reason for this is that bike sheds are easy to design and everyone can have an opinion on it.
In corporate we sometimes spend a lot of time on bikeshedding activities just because our minds automatically go towards simplicity first.
NPS to me sometimes sounds like the Bikeshed of the user research world
As a startup / company I would be more worried about actual referrals, customer churn/ retention, cost of acquisition than NPS. Low NPS maybe a sign of something wrong, but it’s likely also showing up in other survey questions.
NPS may be simple, but not necessarily useful
As an investor I would ideally ignore the NPS score, it’s easy to manipulate and if it’s rewarded it would be in any companies best interests to figure out how to get better scores