Adaptation of my Tweet thread(If the link is broken its likely because my account is private, please follow me and try again)
They say, the key is to not drink the Kool Aid.
While I speak from a more Product manager’s perspective, this is generally true for everyone who wants to be intellectually curious
It’s easy to be be a believer, it gives you sense of purpose, a sense of comfort, but if you really want to succeed, you need to be a bit of a skeptic.
And skepticism can be taught. Here is a simple trick I suggest
Developing a habit of skepticism
Whenever you read or listen to something interesting, something that catches your eye, especially something that makes claims: Think about one small fact that you can verify. I typically add a small note “really?” in my personal notes.
Go ahead and try to see if you can substantiate that. It could be a very simple thing: Eg someone says that a new study says that Covid vaccine is very effective, you can just check if the study exists and it makes that claim. It could be even simpler than that: Eg a startup says that their market is all tax payers in india and that is Y Million people. Just try and find that data, is that accurate.
Slowly you start moving to questioning the interpretation on those facts. Eg: In the covid vaccination is “good” case above, you could now try and substantiate what is good. The research report may say 80% efficacy. Is 80% good? How does it compare to other vaccines. Are there any specific things that have been missed? Which age group, which demographic, which variant?
Eventually you start questioning the whole premise of the argument itself.
At the highest level you move to the very motivations driving the argument.
Most things you evaluate would be correct, but that is not the point. You are not trying to find malice, but just building a muscle for questioning.
With enough practice you would start seeing a pattern. You will get a “gut” for understanding what to check and verify.
Every industry, and also every individual has a pattern of what they tend to overlook or what they exaggerate. It could be personal bias or just a generally accepted “industry practice” (see Uselessness of NPS score article as an example of how a generally accepted industry practice is not necessarily accurate)
With this you are not trying to be cynical, but just being skeptics.
A very good exercise might even be to treat this very article with skepticism
- Have I defined skepticism right
- Can skepticism be taught, or is it just genetic
- Who said PMs need to be skeptics. Is there some kind of qualitative or quantitative evidence to support that claim?
After writing this article, a perfect opportunity to demonstrate this came about. I started seeing some WhatsApp forwards and tweets talking about how upto 40% apple workers intend to leave for lack of full remote, or 90% apple employees want indefinite remote.
While I am all for flexibility and do believe that full remote is here to stay, the 40% / 90% number seemed way too high.
So I decided to look a bit deeper. Thankfully one of the newspaper itself posted all the details including their own skepticism
- This data does exist. It was collected from employee survey done at Apple
- It was done by Apple Employees themselves
- It was done in a slack group specifically meant for people invested in remote work . DUH!!!
Without even trying too hard you can see that only 36% of employees in a group specifically meant about remote work spoke about resigning.
If I was teaching a class on bias, I would use this as a perfect example.
You can draw absolutely NO conclusion about what apple employees want in aggregate from this. While I do give points to media sites for publishing the survey details, I hold them accountable for publishing it in the first place knowing fully well that this data has no validity.
It leads to absurd headlines and unnecessary conclusions amongst people who trust them.
It’s like me doing a survey in a “Board game lover”internal group about how important it is to have board games in the break room, i may get 90% Yes. That does not mean 90% people in my company want boardgames in the break room
NO company is going to lose 40% employees just because the do not offer full remote work. Ironically, the most accurate data about the pulse of their organisation might be with the company itself. The company’s survival depends on it. No one would risk losing 40% of employees.
You can obviously dig further and look at specific questions and see if these questions had inherent bias already. Surveys are not that easy and results can vary widely based on how you ask a question.
Motivation: Now you can ask, why did these news sites publish these results knowing fully well that they are widely inaccurate and biased. What is the motivation:
- They are judged by clicks
- Appealing to bias of people leads to more sharing
- Apple is the current favorite punching bag
Leave a Reply