My reading journey through Thinking, Fast and Slow, Part 2

Over the next year and an half, the North American news media will be inundated with polls. Who leads in what polls and what margin does one have over another. There is nothing wrong with polls, except when we believe strongly in their results without questioning their assumptions.

As the recipients of numeric information, especially when presented as percentages, we should all be aware of the quality of the number. Then, our trust in what the number says should be adjusted accordingly.

Trending

A Reading Guide to Antiracist Books

7 out of 10 is 70%.

70 out of 100 is also 70%.

And so is 700 out of 1000.

They are all equal to 70%, but we know that the quality of the 70% differs. We know (or should know) that larger sample sizes yield more trustworthy results. This is the more intuitive principle from chapter 10, The Law of Small Numbers. The second, however, is less intuitive, although completely equivalent to the first.

***

The best schools are small.

That is an example Kahneman uses in chapter 10. He then summarizes a research finding on the characteristics of the most successful schools. The study concludes that the most successful schools are small in size, which then motivates the Gates Foundation to make large investments to create small schools. Other organizations and government efforts also follow in the effort.

There are many reasons we can postulate on why small schools are more successful. Most of us would find this storyline perfectly acceptable or even subscribe strongly to the conclusion.

Unfortunately, as Kahneman suggests, this analysis is wrong. If the question were asked, what are the characteristics of the worst schools, they would have found that these schools also tend to be small. And as he mentions in another place, a cause that can explain 2 opposite outcomes means nothing at all.

The reason why small schools can produce two opposite and extreme results is because they have small populations. This smallness in numbers yields more variations, i.e., they swing more. You would most likely get a figure like 98% or 1.7% from small samples, not because they are the best or the worst, but because they are small. It is simply a feature of statistics. Any phenomenon that is very high or very low, best or worst, will most likely come from small sample sizes.

***

“But nooo, there must be a reason why small schools perform better.”

Anyone has that small voice speaking in his or her head? I do. There is something inside of me that needs an explanation of this phenomenon.

Kahneman says this is System 1 talking, which likes to find causal relationships to make a coherent belief system, even though some things are mere randomness. These random things can be marked as causal relationships, which lead to a belief that is in fact, completely spurious.

***

The subject of this chapter is not easy to explain and this post is definitely far from sufficient. It takes time to internalize the principles it describes, but its importance is paramount, if you care about the truthfulness of what you think is true.

Is this cause-and-effect that I believe true, or am I assigning causal relationships to something that is, in fact, random?

***

• Always consider the sample size of any statistical study.
• When someone claims a causal relationship, ask whether the same cause can give alternate/opposing/inconsistent outcomes. Then adjust the trustworthiness of that information accordingly.