These are the three questions of the Cognitive Reflection Test:
(1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.
How much does the ball cost? __ cents
(2) If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? __ minutes
(3) In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? __ days
Here's another question for the Cognitive Reflection Test:
In a study of 3,428 university students, 50% of participants across Harvard, MIT and Princeton got a question wrong on a Cognitive Reflection Test. What percentage of MIT students got the question wrong?
Here's ChatGPT's responses. It's fascinating that it provides the same incorrect answers as people... I'm not sure how to interpret this from the robot.
(1) "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total."
(2) "If 5 machines can make 5 widgets in 5 minutes, then each machine can make 1 widget in 5 minutes / 5 machines = 1 minute. Therefore, 100 machines can make 100 widgets in 100 minutes."
(3) "If it takes 48 days for the patch of lily pads to cover the entire lake and the patch doubles in size every day, it would take half that time, or 24 days, for the patch to cover half of the lake."
This post title is heavily editorialized. The article title is "Cognitive Reflection Test - Examples of Reacting vs Checking", and doesn't really attempt to give a canonical answer the question of why many people fail the CRT. It discusses some possible reasons (mainly Kahnemans two modes of thinking) and proposes a 3-step strategy to to avoid falling into the trap of going with ones initial, gut feeling.
The actual reason for the high proportion of failures doesn't seem that complicated, personally I'd label it as "falling for trick questions" and is something everyone is likely to do if such questions come out of the blue, especially if under some time constraint. Our minds tend to be as economical as possible.
The 3 questions quoted in another comment are not trick questions and they obviously did not come out of the blue for anyone taking the test.
They are simple reasoning questions. IMHO, people tend to "fall for them" not because they are trick questions but because people don't really reason carefully.
The standard advice I have always received is to read the actual question carefully, the whole question and nothing but the question.
This might catch a random member of the public but you would think university students are at their peak of "reading test questions skills".
Here I think what the questions do is that they may cause our brain to very quickly come up with a shortcut and wrong answer, which should immediately be discarded if you were indeed carefully reading the question, i.e. paying attention (so maybe indeed something like a conflict between 'fast and slow thinking').
yes exactly, i "failed" that test myself but then i thought, why didn't i do better? Because the cost of failing in this case is absolutely nothing, and the brain automatically optimizes for speed with the first random answer.
I really tried to analyze my thought process before/during/after seeing the problem, trying to solve it, and then seeing the solution.
It seems to me that a lot of our thought process isn't under fully conscious control. We're pattern-matching machines, and math and logic don't come naturally to us. To apply math or logic, you have to train your brain to look at a problem without any shortcuts. Even then, you sometimes need to look at a problem from another angle. For instance, I think that people who read this problem out loud, slowly, may spot the trick more easily than those who just try to reason about it silently in their minds.
Regarding fast and slow:
"It is likely that Kahneman’s book, or at least some of his chapters, would be very different from the actual book, if it had been written just a few years later. However, in 2011 most psychologists believed that most published results in their journals can be trusted."
I asked ChatGPT this question. Here's its first response:
Q: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?"
A: "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total."
Fascinating that it gets it wrong exactly the same way that a human gets it wrong. I kept trying a few times, and eventually could get a correct answer out of ChatGPT. But most of the time it gives the wrong answer.
Specifying MIT et al. makes it look like it’s a specific flaw in stereotypically ‘smart’ people, rather than the general population.
Curious if this is more generally tested and known, or what biases are in place in specific populations. Eg. Bias to immediately trust first thought/gut, vs being taught to be more thoughtful and cautious.
Could be a clearance sale, used sporting goods store, in the past, in another country, etc. It's just not a good question for anything but tricking or frustrating people.
Readit News
(1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? __ cents
(2) If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? __ minutes
(3) In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? __ days
In a study of 3,428 university students, 50% of participants across Harvard, MIT and Princeton got a question wrong on a Cognitive Reflection Test. What percentage of MIT students got the question wrong?
(1) "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total."
(2) "If 5 machines can make 5 widgets in 5 minutes, then each machine can make 1 widget in 5 minutes / 5 machines = 1 minute. Therefore, 100 machines can make 100 widgets in 100 minutes."
(3) "If it takes 48 days for the patch of lily pads to cover the entire lake and the patch doubles in size every day, it would take half that time, or 24 days, for the patch to cover half of the lake."
The actual reason for the high proportion of failures doesn't seem that complicated, personally I'd label it as "falling for trick questions" and is something everyone is likely to do if such questions come out of the blue, especially if under some time constraint. Our minds tend to be as economical as possible.
They are simple reasoning questions. IMHO, people tend to "fall for them" not because they are trick questions but because people don't really reason carefully.
The standard advice I have always received is to read the actual question carefully, the whole question and nothing but the question.
This might catch a random member of the public but you would think university students are at their peak of "reading test questions skills".
Here I think what the questions do is that they may cause our brain to very quickly come up with a shortcut and wrong answer, which should immediately be discarded if you were indeed carefully reading the question, i.e. paying attention (so maybe indeed something like a conflict between 'fast and slow thinking').
It seems to me that a lot of our thought process isn't under fully conscious control. We're pattern-matching machines, and math and logic don't come naturally to us. To apply math or logic, you have to train your brain to look at a problem without any shortcuts. Even then, you sometimes need to look at a problem from another angle. For instance, I think that people who read this problem out loud, slowly, may spot the trick more easily than those who just try to reason about it silently in their minds.
"Costs a dollar more" is not saying that the bat cost $1 as a constant. This was a variable, relative price compared to the ball.
Failure to understand the mutability--or lack thereof--of the symbols in the system is a source of deep woe.
https://replicationindex.com/2020/12/30/a-meta-scientific-pe...
I wonder if the 50% test has been replicated...
Q: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?"
A: "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total."
Fascinating that it gets it wrong exactly the same way that a human gets it wrong. I kept trying a few times, and eventually could get a correct answer out of ChatGPT. But most of the time it gives the wrong answer.
Specifying MIT et al. makes it look like it’s a specific flaw in stereotypically ‘smart’ people, rather than the general population.
Curious if this is more generally tested and known, or what biases are in place in specific populations. Eg. Bias to immediately trust first thought/gut, vs being taught to be more thoughtful and cautious.
Cheapest bat&ball at walmart.com is $10.