No amount of experimentation can ever prove me right; a single experiment can prove me wrong. - Albert Einstein
Thought experiments are experiments that happen strictly inside our heads. The fact that we can test our assumptions about the real world using nothing but our imagination is quite literally mind boggling.
In her wonderful book The Scout Mindset, Julia Galef breaks down 5 types of thought experiments that are designed to help us notice and deal with everyday bias.
The Double Standard Test
The double standard test asks “am I judging other people’s behavior by a standard I wouldn't apply to myself?” To run the test you visualize yourself in the other person's shoes.
I find this test most helpful when reviewing a teammate’s work. By my terrible nature, my initial reaction to any piece of work is seeing all the flaws and ways it can be improved. This could be a design, a product specification document, or a piece of code.
Then I ask myself “would I have done this work any differently?” Instantly, all the hidden complexities, details, and tradeoffs come to the surface. And everything makes perfect sense.
At the end of a double standard test I always have more empathy and respect for a colleague’s work. I treat them a bit closer to how I’d like to be treated.
The double standard test can also help you treat yourself better. Being hard on yourself? Imagine a friend being in your shoes and what advice you’d give them.
The Outsider Test
This one is my favorite.
The outsider test asks “how would someone else deal with the situation I’m in?” It helps you get out of your own head and handle a situation more objectively.
The example from the book talks about the legendary founders of Intel – Andy Grove and Gordon Moore – and how they ended up abandoning the failing memory-chip business which used to be Intel’s identity.
As Moore describes:
Our mood was downbeat. I looked out the window at the Ferris wheel of the Great America amusement park revolving in the distance, then I turned back to Gordon and I asked, “If we got kicked out and the board brought in a new CEO, what do you think they would do?” Gordon answered without hesitation, “They would get us out of memories.” I stared at him, numb, then said, “Why shouldn’t you and I walk out the door, come back, and do it ourselves?”
I find it extra helpful to imagine someone I admire coming in and taking my role. How would Steve Jobs approach the situation differently? I bet he wouldn’t sit there and whine about it as much.
The Conformity Test
The conformity test asks “how strongly will I hold this opinion if everyone else changed their mind?” It helps you understand if your “own” opinion is actually your own or someone else’s.
Oftentimes what feels like our independent opinion is nothing more than us conforming to some else’s point of view. When agreeing with someone else’s viewpoint, do a conformity test: imagine that this person told you that they no longer support this view. Would you still hold it? Would you go and defend it to them? Or will you change your mind in 2 seconds?
I find this test most valuable with people I trust deeply. Trust is a double edged sword, it can be the foundation of healthy conflict but can also lead to conformity.
I do this with my wife and my co-founder a lot, which is why neither of them likes me that much.
The Selective Skeptic Test
The selective skeptic test asks “if pointed in a different direction, would I weigh this piece of evidence differently? ” It helps you realize when you’re cherry picking evidence to support your viewpoint.
A good example is user feedback:
You have an idea for a feature
You test it with 10 users
3 users confirm that it is very valuable to them
You build the feature based on the story of those 3 users
What about the other 7 users? “They were not the right users to ask. They have a different use-case. They just didn’t get it. They are not as credible.”
The selective skeptic test goes like this: Imagine the evidence supporting the opposite case. How credible would you find it then?
In other words, if those 3 specific users did not find the new feature valuable, would you still blindly follow their input and discard the other 7 as being non-credible?
The Status Quo Bias Test
This one is also my favorite. It is best explained by examples.
Let’s say you’re trying to improve a feature that is not being used. If the feature didn’t exist at all, would you explicitly prioritize it and put time into building it?
Let’s say that you’re suddenly unemployed, would your current job be the first and only position you’d apply to?
Let’s say that your current relationship suddenly did not exist, would you do everything in your power to get it started?
If the answer is anything less than a resounding yes then you’ve got some soul searching to do.
Conclusion
A thought experiment is a power tool for being less wrong and less biased. We tap into our imagination, look at the situation with a different lens, and come back seeing reality better.
We’ve covered 5 thought experiments:
The Double Standard Test - To judge people fairly, imagine how you would behave being in their shoes. Would you judge yourself differently?
The Outsider Test - To better assess a situation you’re in, imagine that someone else was in this situation. How would they act?
The Conformity Test - To test whether your “own” opinion is indeed your own, imagine that other people no longer held it. Would you still defend it?
The Selective Skeptic Test - To weigh evidence fairly, imagine that it supported the other side. Would you still count it or perhaps discard it?
The Status Quo Bias Test - To understand your own situation objectively, imagine it no longer being the default. Would you actively choose it?
Don’t believe in thought experiments? Try to imagine how they could improve your life.