While I speak from a more Product manager’s perspective, this is generally true for everyone who wants to be intellectually curious
It’s easy to be be a believer, it gives you sense of purpose, a sense of comfort, but if you really want to succeed, you need to be a bit of a skeptic.
And skepticism can be taught. Here is a simple trick I suggest
Developing a habit of skepticism
Whenever you read or listen to something interesting, something that catches your eye, especially something that makes claims: Think about one small fact that you can verify. I typically add a small note “really?” in my personal notes.
Go ahead and try to see if you can substantiate that. It could be a very simple thing: Eg someone says that a new study says that Covid vaccine is very effective, you can just check if the study exists and it makes that claim. It could be even simpler than that: Eg a startup says that their market is all tax payers in india and that is Y Million people. Just try ad find that data, is that accurate.
Slowly you start moving to questioning the interpretation on those facts. Eg: In the covid vaccination is “good” case above, you could now try and substantiate what is good. The research report may say 80% efficacy. Is 80% good? How does it compare to other vaccines. Are there any specific things that have been missed? Which age group, which demographic, which variant?
Eventually you start questioning the whole premise of the argument itself.
At the highest level you move to the very motivations driving the argument.
Most things you evaluate would be correct, but that is not the point. You are not trying to find malice, but just building a muscle for questioning.
With enough practice you would start seeing a pattern. You will get a “gut” for understanding what to check and verify.
Every industry, and also every individual has a pattern of what they tend to overlook or what they exaggerate. It could be personal bias or just a generally accepted “industry practice” (see Uselessness of NPS score article as an example of how a generally accepted industry practice is not necessarily accurate)
With this you are not trying to be cynical, but just being skeptics.
A very good exercise might even be to treat this very article with skepticism
Have I defined skepticism right
Can skepticism be taught, or is it just genetic
Who said PMs need to be skeptics. Is there some kind of qualitative or quantitative evidence to support that claim?
After writing this article, a perfect opportunity to demonstrate this came about. I started seeing some WhatsApp forwards and tweets talking about how upto 40% apple workers intend to leave for lack of full remote, or 90% apple employees want indefinite remote.
While I am all for flexibility and do believe that full remote is here to stay, the 40% / 90% number seemed way too high.
So I decided to look a bit deeper. Thankfully one of the newspaper itself posted all the details including their own skepticism
This data does exist. It was collected from employee survey done at Apple
It was done by Apple Employees themselves
It was done in a slack group specifically meant for people invested in remote work . DUH!!!
Without even trying too hard you can see that only 36% of employees in a group specifically meant about remote work spoke about resigning.
If I was teaching a class on bias, I would use this as a perfect example.
You can draw absolutely NO conclusion about what apple employees want in aggregate from this. While I do give points to media sites for publishing the survey details, I hold them accountable for publishing it in the first place knowing fully well that this data has no validity.
It leads to absurd headlines and unnecessary conclusions amongst people who trust them.
It’s like me doing a survey in a “Board game lover”internal group about how important it is to have board games in the break room, i may get 90% Yes. That does not mean 90% people in my company want boardgames in the break room
NO company is going to lose 40% employees just because the do not offer full remote work. Ironically, the most accurate data about the pulse of their organisation might be with the company itself. The company’s survival depends on it. No one would risk losing 40% of employees.
You can obviously dig further and look at specific questions and see if these questions had inherent bias already. Surveys are not that easy and results can vary widely based on how you ask a question.
Motivation: Now you can ask, why did these news sites publish these results knowing fully well that they are widely inaccurate and biased. What is the motivation:
Sometime back I had tweeted a thread about how I dislike NPS score as a measure of success. It led to a fair amount of discussion and debate. Hence, I decided to dig a bit deeper and write a longish post about it. While I have tried giving it some structure, each section is more or less self contained. You can directly jump into a specific section if you are already aware . I have tried to give enough context wherever possible.
I have also linked to sources wherever applicable, so feel free to follow them and do your own due diligence when in doubt.
Disclaimer: I may add more details and address any specific questions and criticisms that may come my way. Please do let me know if I misrepresented or missed something
What is Net Promoter Score AKA NPS
Net promoter score is a widely used measure of customer loyalty today. It’s claim to fame is its utter simplicity. It can measure customer loyalty with just 1 question.
To calculate Net Promoter score you ask a statistically significant number of your customers to answer a single question
How likely is it that you would recommend [brand or company X] to a friend or colleague?
Ask them to select from a scale of 1-10(some orgs use a slightly different scale but 1-10 is the most widely used), where 1 means not likely at all, 10 means very likely and 5 means neutral.
Anyone who choses 1-6 is considered a Detractor
Anyone who choses 9-10 is a Promoter
Net Promoter Score= %age of Promoters – %age of detractors
The basic claim of NPS is that it can reliably measure customer loyalty and if the company focuses its efforts to increase NPS, it can lead to more healthy growth.
This two specific claims are important to keep in mind 1) Reliably measure loyalty (better than other scores) 2) Correlated to company growth (see more details in NPS Origin story sec)
NPS origin story
While internet is filled with how to use NPS, when to use NPS, and why to use NPS, before we get to all those questions it is necessary to understand how NPS even came into picture.
The origin story of anything reveals a lot about the motivations without the burden of muddled history between then and now.
NPS score was invented by Fred Reichheld who was a consultant at Bain and company . It was introduced to the world in this HBR article
The basic idea came about when they looked at a car company use a very simple method to increase customer loyalty. The company, Enterprise Rent-A-Car, simply asked people two questions – Quality of rental experience – Likelihood you would rent again
The company then counted only those customers who gave it the highest scores on both the questions. All their outlets were then asked to optimize for this specific score. It was believed that this would inspire the sales agents to be better and increase customer loyalty.
Fred wanted to make this system much more simpler and see if this could be brought down to just 1 question
The interesting point to note here was that the intent was not to find a great predictors of company growth or loyalty, rather to find one question. The aim itself was simplicity
Process of finding the One Question
20 questions were created on the Loyalty Acid test survey
Test was administered to customers in following industries
Cable and telephony
Internet service providers
Then they asked each participant to tell about a specific instance when they actually referred the company to someone. If this was not available they waited 6-12 months and asked again. This data of about 4000 users was enough to create 14 case studies which established a link between survey response and actual referral
The top-ranking question was far and away the most effective across industries:
How likely is it that you would recommend [company X] to a friend or colleague?
Two questions were effective predictors in certain industries:
How strongly do you agree that [company X] deserves your loyalty?
How likely is it that you will continue to purchase products/services from [company X]?
Other questions, while useful in a particular industry, had little general applicability:
How strongly do you agree that [company X] sets the standard for excellence in its industry?
How strongly do you agree that [company X] makes it easy for you to do business with it?
If you were selecting a similar provider for the first time, how likely is it that you would you choose [company X]?
How strongly do you agree that [company X] creates innovative solutions that make your life easier?
How satisfied are you with [company X’s] overall performance?
Link between NPS score and company growth
Then they tried to find correlation of NPS score of customers with the actual company growth
In airlines a strong correlation existed between “Would recommend” question and average company growth
Similar results existed in rental car business
“Would recommend” was irrelevant for database software or computer systems as people had limited choice, and senior execs who made the choice were not part of the people surveyed. For such industries “Sets standards of excellence” and “deserves your loyalty” were far more predictive
NPS was also not a predictor for Local telephone and cable TV company growth because they were near monopolies. Their growth was determined by how fast the population in their area increased
Who uses NPS today
Pretty much every one. As of 2020 2/3 of fortune 1000 companies seem to use a version of NPS. One simple experiment would be to search for the term “How likely” in your Inbox
Good things about NPS
It is very simple to measure and benchmark .
Its a single question and is used by multiple industry players to benchmark against competition and internally
Its easier to digest at almost all levels of abstraction
High completion rate
With users being inundated with all kinds of brands seeking their attention, it is much more easier to get them to answer 1 question rather than multiple. Infact in the paper “Assessing treatment outcomes using a single question” where they did an NPS of patients, they found that the NPS question consistently had the highest completion rate (96.5%). I would also now assume that it has become so common that users almost expect this question and are willing to answer
It defines loyalty in interesting way:
While loyalty may traditionally be defined by retention, LTV, and other metrics , it can miss out on word of mouth. NPS attempts to target that specifically by a bit lose but interesting way
Customer Loyalty Definition in Original NPS system
Customer loyalty can be defined as customers willingness to stick to certain provider even if they are not providing the best possible rate in a particular transaction. Think if this like : ” Sure you may be charging me more today, but I know you have done great work in the past and generally give me good rate so I will stick to you even though cheaper options may be available”
Customer loyalty is also more than just retention because some people maybe retained just because they cannot move out due to inertia, or exit barriers. Eg : Monopoly players , or prepaid plans
Loyal customers may also not be repeat purchasers, eg when they outgrew that service. Eg: You may no longer buy a pulsar bike because you no longer drive a bike, but you would recommend it to your nephew when he is considering one.
NPS claims and how they measure up
NPS is a slightly obtuse metric because instead of asking if people are satisfied with the product or service, we are asking if they would recommend it to someone else. It’s not exactly a measure of a customers own experience with the brand.
If you are introducing a new kind of measurement it needs to be better at something than the existing systems. It either helps you uncover a specific issue, or measure something unique.
Survey metrics also are predictors / proxies of some tangible business outcomes such as churn, growth, complaints, etc. A metrics with no business outcome is plainly a vanity metric.
So lets deep dive into if NPS measures up
NPS as a better predictor of growth
Let’s look at the claims made about NPS in its original research. There are multiple leaps of faith in it. The way I read the original article is:
Answer to NPS question seems to be the highest correlated among other questions to actual referrals in some industries
Higher NPS seems to be correlated to higher growth rate irrespective of company size
Using this above methodology claims have been made that NPS is the best predictor company growth. The big issue with this is that even in the original article there was no real comparison of correlation between company growth with NPS vs other survey methods.
Also even though the question seems to talk about loyalty in a very loopy fashion, it actually does not make a claim about it. There may or may not be no correlation between NPS and user retention
This research was not even reproducible
It is not reproducible
This is perhaps the BIGGEST issue with Net Promoter score. The biggest claim with NPS was it is the single best predictor of growth, but this 2007 paper found no support for that claim when they tried to replicate the same study that Reichheld did.
Not surprisingly, they found that NPS performed as well or as poorly as using the customer satisfaction index to predict growth
There seems to be no real statistical backing to NPS, and as per the paper, even Reichheld acknowledges that
NPS as a tool to benchmark competition
A lot of literature outside talks about using NPS as a benchmark against competition, between different departments, different franchise etc. A lot of fanfare is made about how a company’s NPS is through the roof, which company in a specific industry has the highest NPS etc.
The problem with this is that this question has so many variables that its unfair to compare . It can never be an apples to apples comparison.
Instead of simply asking if customers are happy with the service, we ask “Would you recommend X to your friends and coworkers”. There are so many more variables to consider when trying to answer this question
Do I think it’s worth it for my friend: Hobbies, cost, personal interest, my own closeness to the friend
Do I even discuss this with my friends and coworkers
I hate it, but my this specific friend may like it
More variables = more errors.
Eg: when NHS introduced NPS they found that only about 40% variations in NPS scores was explained by overall satisfaction whereas rest was explained by various other metrics such as: if the patient undertook hip replacement or knee replacement.
Even the NPS difference between patients who underwent Hip replacement(71) and knee replacement(49) were stark, making it impossible to benchmark them.
A bad action item for the hospital would be to target for same NPS across all services.
If a hospital cannot even benchmark within its own departments, it’s useless to try and benchmark to other hospitals.
It’s also very dependent on services availed and demographics of the user . Eg: when they compare NPS of Uber vs Ola, they fail to talk about if it’s the exact same mix of users or not. Did the users take similar number of trips, same kind of vehicles , pool vs non pool etc. Without that, comparing NPS of brands Uber vs Ola is not really any benchmark, and potentially worse than just satisfaction surveys. It’s not a worthy complication you are introducing. This is why brand NPS are not really benchmarks
For internal benchmarks, some companies, especially in ecommerce, go overboard and try to find NPS linked to each product and service, which again seems unnecessary. It is no longer a measure of loyalty but just a feedback of the product, which may be better asked directly via ratings and reviews.
Eg see below Myntra trying to do NPS linked to a specific product. But even if I rate it low they may have no action item because they do not control the product itself. Lot many questions need to be asked to even understand my response. A better question would have just been do you like the product, which not only would be direct but also feed their rating system.
NPS as a better loyalty metric
This is another way some people use NPS for . This is potentially because of the nature of question where it talks about referrals. Loyalty here is defined as user’s willingness to recommend.
NPS tries to force fit people into specific boxes of promoters and detractors ignoring any reasoning behind the users response
There are many reasons you would not recommend a specific product to someone. Just like in NPS article they mentioned that a loyal customer may not be a repeat customer because they outgrew the product, but would happily recommend it to someone who did not.
Using the same logic, someone who may be a loyal customer may not recommend it to friends who may not be the target customers. NPS would classify these loyal customers as detractors.
Loyalty is very fluid and detractors and promoters are not rigid boundaries as NPS tries to bucket them i
Your personally may hate the product but if you were to suggest a product to someone you would play a matchmaker role and take into consideration their individual needs and circumstances.
This 2019 survey found that 52% users who actively discourages others from using a brand also actively recommended it. You can be a promoter and detractor at the same time based on who you are talking to or how your last experience with the brand has been.
To make matters even more complicated, NPS is not even an accurate predictor of users own measurable behaviour such as repeat purchase, churn rates etc
The purest measure of loyalty in my opinion is customers actually spending money to buy your product. I criticise my bank a lot, but despite many many alternatives I have stuck with them for 15 yrs. By every definition, I am a loyal customer who they would want.
Another argument for use of NPS is that its less susceptible to manipulation, but NPS has same pitfalls and anyone who owns an NPS goal can use the same old tactics to improve it.
Just like satisfaction surveys which can be manipulated, so can NPS. Simple techniques could be
Asking the user to rate you after a good interaction. Eg: As soon as order is delivered, or a ticket resolved. At this time you are no longer trying to find and fix issues, you are simply trying to get that score. This technique may have an effect for Play store reviews and youtube videos where ratings and likes are a social signal to other users, they are counter productive for NPS , unless this NPS is being collected as a vanity metric. Eg: Pitch deck, Presentation to leadership
Incentivising the user: Eg give your software product for a free trial and see almost every customer give you high ratings on NPS. It is meaningless and may have no correlation to your growth
I also did a very unscientific survey on Twitter and Linkedin to know if companies took NPS targets, and if the person responsible for the target also controlled stuff like when NPS was sent and how to pacify the user: Here are the results
As per the survey above ~30% respondents said their company has NPS goals and the owner of the goal optimises of things like when to send the survey, and in some cases even customer incentive . You get what you optimise for and in this case my hypothesis is that system is designed to make the NPS go up not necessarily the loyalty
You get what you optimise for
This maybe the reason why companies with really high NPS also go bankrupt
NPS and other big misses
Its arbitrary and ignores all cultural nuances
There seems to be no clarity on why someone who says 6 vs 7 are in a different bucket while 5 vs 6 are not. Also no clarity why focussing only on difference between promoters and detractors matter. What if we just tried increasing average score?
It actually hides the actual improvements. Eg: movement of a large chunk of users from 1 to 5 has no effect on the NPS score.
It also ignores all cultural nuances. Eg: if you travel in Uber in US vs India, you may see a huge difference in your ratings. Anecdotally I have seen my ratings drop in india and rise in US. I presume there is a cultural difference here. In India low rating is 1* whereas in US its 4* .
I read a comment on some blog that put it well: NPS is just lots of numbes disguosed as maths
It is more noise than signal
What do you do with NPS? One common theme is that you work towards increasing it by using it as a north star, but that is not a good reason to ask this question in the first place. There is no evidence that it is better than working to optimise other tangible metrics.
It’s not a single question
While the whole USP is its single question, you invariable would need more information as soon as the users rate <7 , defeating the whole purpose of simplicity.
Loyalty is multidimensional
While NPS seems to acknowledge that loyalty is multi dimensional, it tries to collapse it into a single dimension of word of mouth.
Its probably not for your industry
This is less of NPS issue and more of marketers abusing NPS because of its perceived simplicity.
In the original paper, NPS was not found to be a predictor of growth in industries such as computer databases.
Remember the ONLY thing it was supposed to do was predict if you will grow, without that correlation the score is more or less useless.
Sales is complex and any industry with high inertia, top down decision making, and monopolistic players NPS is not even applicable. This makes me wonder why so many startups are obsessed with it.
It’s also possible that NPS should not even be a goal.
Eg in the NHS paper I referred to, difference in NPS among patients was not due to actual patient care and recovery. Perhaps NPS is not even a measure for hospitals.
NPS seems to be a arbitrary score with little statistical backing. It is not even be valid for many industries . While it can be used as a tool in your armour of many other signals, over reliance on this for making decisions is not prudent.
NPS is popular perhaps because it is simple, but this reminds me of the phenomenon of Bikeshedding .
Bikeshedding: If a committee were to design a nuclear power plant, they may spend far more time than necessary to discuss the bike sheds, its color, its position, and its capacity . The reason for this is that bike sheds are easy to design and everyone can have an opinion on it. In corporate we sometimes spend a lot of time on bikeshedding activities just because our minds automatically go towards simplicity first.
NPS to me sometimes sounds like the Bikeshed of the user research world
As a startup / company, I would be more worried about actual referrals, customer churn/ retention, cost of acquisition, than NPS.
Low NPS maybe a sign of something wrong, but it’s likely also showing up in other survey questions. NPS may not be adding any value
NPS may be simple, but not necessarily useful
As a product manager, I become very suspicious when some startup or product touts high NPS scores with little else to back it up.
As an investor, I would ideally ignore the NPS score, or give it very less weightage unless backed by actual metrics. it is easy to manipulate and if it’s rewarded it would be in any company’s best interests to figure out how to get better scores.
As some of you already know, I am LONG twitter and feel that it has just started to just scratch the surface of what is possible. Recently twitter announced a slew of new features including
Spaces : Clubhouse competitor
Subscriptions: Subscribe for
Exclusive tweets / content
In January twitter also acquired revue, a newsletter company to let twitter users start newsletters
They also announced that they are reopening “Verification” in early 2021 and users would be able to request for it
Where is it going
I beleive twitter is unleashing the beast with the power of its platform and could eventually go into Identity and Social capital management. Here are some of my early thoughts
Twitter as Identity Management
Once twitter starts mass verifying users, it would become perhaps one of the largest consumer facing identity management platform in the world. It would be bigger than many governments and will come with added social capital.
When someone says they are verified, you not only know who they are, but can also go and check out their profile. There is a massive need for user identity verification especially in situations involving people to people contact. Eg Uber users in brazil need to validate via CPF(national ID ) if they want to pay via cash. This helps them establish trust. Think about using twitter globally
Other user cases could be
Is the person sending me the email the same as they claim to be
Is the person bidding for my furniture real
A lot of startups bootstrap on twitter network( Eg: Substack), imagine what could be done with high trust authentication.
Twitter as Social Capital Management
Whenever any influencer or content creator opens up another channel (Eg a new newsletter, a youtube channel, a blog) they end up recreating the entire network again. The need to gain the trust of not just the followers but also the platform.
While there surely are massive advantages of existing audience which follows the creator, there is always a huge leakage and the creator is subject to very different rules on each.
Eg: Even if you are extremely popular on twitter, you cannot take that social capital to say a mailchimp to send your newsletter. You need to “warm up” the system, hope your audience opens it, prey that spam filters don’t flag you, and if you have a shared IP hope all other providers are not doing something shady.
Mailchimp has no context of who you are or how you built your audience. There is no social capital associated with you.
But with twitter this could change. Twitter could use your social capital of one platform (say twitter) to jump start you on another (eg newsletter). It can use your social capital for more than just growth hack.
Will it matter to small time creators
While small social capital on one mode of communication may not matter to people, a multi modal platform may matter in aggregate.
The biggest advantage of this is that as people start deriving more value from the system, the incentive for bad behaviour becomes less You also become super cautious if you verify at any place using twitter because you don’t want to mess with your social credit.
Gotchas to watch out for It needs to be debated if it would be wise for a single private party to wield such power. Would your twitter identity become more important than your local identity?
Some interesting ideas I think twitter would / should do
More control on tweet embedding: Let me generate a id that any media agency that wants to embed my tweet needs. I can then charge for my tweets
Transfer tweet ownership: Currently if your tweet becomes popular, people reach out to you to add their product / business as a reply. This helps people make some money. It would be interesting to see if I could just pass on the tweet ownership to someone else and they would be free to add any reply to the main thread
NFT: While some startups are attempting this. Every tweet could be an NFT and that can drive the tweet embedding and ownership ideas as well.
Tweet Ads: Every time a tweet is embedded on an external site, twitter could show and ad and share revenue with the tweeter directly
Mail client: While twitter knows about the social capital, it still is hindered by the spam rules of other mail client providers. I suspect they may start their own
Influencer Management / Tracking / smart contracts: Imagine you hire an influencer for reaching X million audience. The entire deal is done on twitter where twitter is responsible for measurement of success (which they already do). As soon as the success criteria is met, the influencer is paid . This operation could be controlled via ethereum smart contracts. This makes the entire system seamless and painless
While we would like to be good at everything we do, it’s neither possible, nor an attempt advisable.
You will always be bad at something either due to skill or by choice. I want to talk about the latter today.
Being bad by choice: Your profile, your CV, your choices, are all reflection of what you want to be. While you want to be strong at what the job you want to do requires, it maybe a good idea to be bad or be perceived ‘not great’ at stuff you don’t want to do.
For example: when I was in consulting, my company offered to pay for PMP certification for any consultant willing to take it. PMP is almost an industry standard for becoming a project manager.
While it rightly got a few people excited, I blatantly refused to do it. I am not against being good at project management, I feared I would suddenly become eligible for project manager roles that may need urgent fulfilment within the company.
It’s super hard to say no early in career and I specifically did not want to be a project manager. I did not last long as a consultant either, but that’s a story for another time.
Doing so saved me from two things:
I knew if the need came, no team looking for project managers could force me or my manager to let me take that up. I preferred bench over Project Management.
A lot of Product Management roles in India are disguised project management roles. Due to my lack of PMP I was not considered as a candidate for them. I auto rejected myself
Traditionally, we assume, amassing certifications and varied skills increases our likelihood of landing a job, but it can also reduces our chances of landing the one we want.
Just because it’s free knowledge does not mean it’s good for you. You need to opt out of and close doors you don’t want to be tempted or forced to enter.
Will I ever do a PMP? : Never say never. I have deep respect for Program managers who can handle large projects and keep seeking guidance and help from them, but if I do take up formal certification I’ll most likely not tell people about it :).
I am at peace with being labelled as a Product Manager who is great at Design, Research, and Data, but sucks at Project Management. I don’t think that has hindered my career yet.
Edit button on twitter is possibly the most requested feature in any product EVER, but somehow twitter never really introduced it. It is also possibly the only “Social Media” where you cannot edit as soon as you post. You can only delete and repost or add a correction in reply.
The feature is so highly requested that even twitter trolled the world by saying, you get an edit if everyone wears a mask. Great way to spread the message but basically saying, NAAAH
My Take is that
Twitter will likely NEVER give an edit button. It’s not a simple social media. Tweets are public, they are quoted, and become news. Tweets are a system of records of a person’s opinion
Tweets are a system of records for a person’s opinion.
There are no edits in it, only new versions. All history is maintained in perpetuity unless it’s useless. Each person is aware of exactly which version of persons opinion they interacted with.
Ideally they should not have delete but thats survival, you probably wont use it without an ability to delete.
I suspect if ever twitter decides to allow some kind of edit, it will come up with some kind of versioning system for tweets. Each interaction would be tagged to the exact version. 10 comments on version 1, 20 likes on version 2 But most likely that wont happen anytime in the near future.
While as a user I hate not being able to edit, I can clearly see its utility. This system of record is so powerful that you now have an authoritative view on what the person thought in the past. It’s either authoritative, or it has been deleted. There is no guessing.
Its a phenomenal space to be in.
Twitter is more valuable because it does not have an edit
I see the value so much that If Twitter ever added an edit button like Linkedin or FB, I will probably sell whatever Twitter stocks I hold
You cannot and should not build every feature that is there in your backlog. One way many companies, especially startups, gauge the need of a specific feature is via a Fake door test.
What is a Fake door test
To put it simply, instead of building the feature your product simply pretends to have that feature and shows user a Call to Action to use that feature. When the user clicks on this CTA, you either tell the user that this feature is not yet ready and you have noted their interest, or tell them how they can use alternatives ways for completing their tasks.
Example: You are a Product Manager at twitter and twitter introduced a way to animate your tweets. This feature is only available on the app and you are trying to gauge user interest on desktop.
Instead of building this, you simply add an “animate this tweet” button on desktop and if a user clicks on it, you can tell them
” This feature is not currently available on desktop but you can always go too the app and use it, click here to see how”
How does this help
You get data on potential size of the market
You know which segment of user needs this feature
You can validate almost all potential new features
Isn’t this disingenuous to the user?
It may seem so, but your job is to solve the most important needs of the user. If this helps you prioritise better, it is a win win for everyone. Sure some people will be a bit pissed but will be overall better served if you build the right features.
Some things to keep in mind
Make sure the call to action is super duper clear. You do not want to be in a situation where users may get confused. Example: in Uber if we wanted to test if people wanted to schedule a ride on the web, giving users a CTA such as “Schedule a ride” would have been misleading because Schedule may mean, schedule for NOW. So a good CTA might be “Schedule for future”. You can work to fine tune the verbiage later
Run this as an A/B if possible and only for a very small segment. You do not want your entire customer base pissed off
When users click on the fake door make sure you explain it well. Don’t leave them hanging. Even better, if you do end up building this feature, maybe send a note to these users
Measure effect on other interactions as well. You need to know the unintended consequences
It is perfectly ok to not run a A/B if you don’t have the infra or enough traffic and collect data only for CTA interactions. But be clear about that miss
Make it dirty and cheap. Don’t over think. Don’t end up doing user studies on what the fake door should look like. You are looking at directional results and your outcome should ideally be “Yes we are building this”, or, “No we are not”
Fake Door vs User Study and Surveys
User study is an awesome tool and can really help you understand the customer better. It also helps you test out multiple things at once.
Fake doors are super helpful when you have a very specific thing to know. While you can also run surveys to gauge interest in certain features (Eg: survey to find if the new integration you are thinking of is valuable to the user), going by the old adage goes “don’t listen to what they are saying, but look at what they are doing”, Fake doors can be supremely helpful and surprisingly accurate even when not stats sig
Fake Doors for Validating Startup Ideas
This seems to be common. I have read multiple accounts, and have seen multiple examples, of people who setup simple websites with a real looking product page, and when you click on “Buy” or “Subscribe” they tell you its not yet ready but you can enter your email ID for early access. If you gather enough email IDs, you have a winner idea and its time to Build
In product management and In general one powerful weapon in your kitty is the ability to organise ideas and features in a logical structure. It helps you think better and in a more focussed way
My favourite being (the one I use) the flow of information. It has 3 parts
1: Creation 2: Curation 3: Consumption
You can literally fit hundreds of things into these buckets
Eg : You are at Twitter and you want to improve Twitter
Think creation – how to get more people to tweet – how to get people to tweet more – more ways to tweet – what apart from tweets
Think curation – hashtags – lists – personalisation – follow – organize in replies and threads – curated bookmarked tweets – trending – add metadata to tweets -og tags
Think consumption – how to view threads – different devices and surfaces – read me tweets – api integrations to send tweets to external providers – surface og tags,images etc – tell me why am I seeing this
Sounds pretty simple, because it is. Breaking down into simple structures can help you focus your attention and prevent you from getting overwhelmed
Some unpopular opinions I hold (Subject to change, of course) -Tech Corporate world
Presentations are mostly useless. They waste time, encourage intellectual laziness, are manipulative by nature, and lead to bad decisions.
Open offices are bad for both productivity and collaboration. They mostly increase distractions., and any gains in collaboration are small in comparison to the losses of productivity.
Almost every tech worker who can work from home will benefit from doing it once in a while. There is no one who works best ONLY in office. Companies should actually force employees to work from home once a while, this forces them to create a good “working corner” at home, and can actually be better for their mental health.
Forcing an early start time for your company is bad for business. If you treat your company like a school, you will get school children to work for you.
Culture is almost always top down. Bottoms up is only feedback and can do very little to actually move the needle.