Is it? That's the very basis for error that begat this article. In fact, Turing's and Kahneman's minds were too open. They didn't express sufficient reservation by demanding more rigorous tests of those claims.
Perhaps a better maxim would be, "It's good to have a mind that's open just enough to entertain the impossible."
That saying, or ones like it, are also attributed to Carl Sagan, G. K. Chesterton, James Oberg, Arthur Hays Sulzberger, Virginia Gildersleeve, Bertrand Russell, Max Radin, ...
This article mentions Walter M. Kotschnig as the author of the earliest known instance of the saying, but notes that even Kotschnig seemed to be quoting something that was already known at the time:
An appropriately simplistic quote from someone who habitually and sometimes famously demonstrates being stubbornly narrow minded. But then, how else would he not know that brains don’t fall out of—nor are placed in—minds.
In addition to statistical evidence people should really examine the plausibility of the mechanism in question. Most accounts of ESP have no description of how it might actually work. The theories are typically dualistic, vitalistic and even plain magical.
What kind of “mechanism of action” you are looking for? I mean, physicists provide no “mechanism of action” for the forces they stipulate to exist. There is no “mechanism of action” for electromagnetism, it just does what it does. Similarly, ESP force might not have “mechanism of action”, it just could do what it does.
What you are doing here is that you are betraying two huge hidden assumptions that you implicitly make. First is that you are most likely expecting to obtain “mechanism action” in some kind of mathematical terms, which implicitly assumes ability to use mathematics to explain the observed universe. Second is that you are looking for “mechanism of action” for ESP in terms of known physical forces and theories. Here, you implicitly assume that known physics is all there is.
Now, I do not believe in ESP either, my point is only that you are demanding rigor from ESP you don’t from electromagnetism.
imo having an open mind means rather exclusively to allow the possibility to be moved by scientific evidence. Not just believe for the sake of it or because of some kind of bias If we don't keep an open mind, in this sense, we'd still believe the Sun rotates around the Earth.
> People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psvchological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions.
(emphasis mine)
Not only does the statement say that “people” make this mistake then go on to cite a questionnaire only of professional psychologists, but the questionnaire is the prototypical non-random random sample the very psychologists taking it are allegedly proving misinformation about. Questionnaires select for people with the time, inclination, and attention to start and then finish a questionnaire.
Infamously, experiments held at universities bias towards undergrads. Experiments that reward participation bias towards people motivated by the reward. Experiments that don’t reward participation bias towards people good-natured enough to contribute just for science.
Certainly very ironic. But is this line of argument incorrect?
E.g. if I said "I don't believe in the Holy Book because in verse 7 it says that one cannot trust anything written in any books". Isn't that an analogous reasoning?
I think these examples/arguments are ultimately about exposing a liar-paradox statement, and when you can show such a statement you have proven that something isn't right.
Yes. And might I say everything has a statistical bias. Picking people for an experiment is not like throwing dice
Your sample will always be biased, and it would be crazy to try and get a random sample from "the world". (and even then you have temporal bias - you can't run your experiments on people from the XIX century)
> The idea you should focus on, however, is that disbelief is not an option.
The results are not made up, nor are they statistical flukes. You have no
choice but to accept that the major conclusions of these studies are true.
More important, you must accept that they are true about you.
So so arrogant, and he ended up being wrong about it too. It's very hard to take him seriously after reading this.
Yes. It is. But as another poster has noted, he largely recanted in 2017 [1]. And this is to his credit. Few scholars -- and fewer prominent ones -- have been willing to reverse course in as public a fashion as this.
“The argument is inescapable: Studies that are underpowered for the detection of plausible effects must occasionally return non-significant results even when the research hypothesis is true – the absence of these results is evidence that something is amiss in the published record. ”
> I am still attached to every study that I cited, and have not unbelieved them
So he admits that the studies don't replicate but he still believes in their conclusions? Somehow that makes him look worse, not better, in my eyes but maybe I'm too critical.
I think you're somewhat misinterpreting his point. He's not saying "I'm correct whether you like it or not". Instead, he's saying "we are all prone to cognitive errors, whether you like it or not".
Admittedly, the experiments he conducted and cited were found to be flawed. But I think his point, as I understand it, still stands.
It sounds like he's saying his students were wrong to express disbelief over the counter intuitive findings of a study because the statistics were such that it left no option but to accept the results. Which is arrogant and wrong. Studies can be faulty or fail to replicate, statistics can be misapplied or fudged, future studies also contradict prior findings, etc.
Blind faith in the results of a study is not good science. If a study is counter intuitive, one should question it.
The point doesn't stand, at least, not for psychology papers: even if the data have been collected correctly, the method was flawless, and the statistics have been done properly, the conclusions (as written in the paper under "Conclusions" heading) are always beyond the scope of the experiment, often far beyond it. I've never seen a psych paper that limits itself to just the experiment. They always present the outcome as proof for some kind of broader theory (which usually has been proven wrong in the same style by somebody else).
> My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young
people on the job market. Being associated with a controversial and suspicious field will put them at a severe
disadvantage in the competition for positions. Because of the high visibility of the issue, you may already
expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old
enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the
replicability of findings: subliminal perception and dissonance reduction.
> I believe that you should collectively do something about this mess. To deal effectively with the doubts you
should acknowledge their existence and confront them straight on, because a posture of defiant denial is selfdefeating. Specifically, I believe that you should have an association, with a board that might include
prominent social psychologists from other field. The first mission of the board would be to organize an effort
to examine the replicability of priming results, following a protocol that avoids the questions that have been
raised and guarantees credibility among colleagues outside the field.
I'd say that it wasn't the epistemology that was flawed, but the studies. (Though, ok, arguably it was a flaw in the epistemology that it didn't allow for flaws in the studies.) But the more interesting question is how the initial studies could have been so flawed. At any rate, those results were not replicated, and science marches on.
Yeah, your reaction is. The whole point was to be humble and accept that you might be wrong. In particular, if well-done research contradicts your prior belief, then you "have no choice" but to consider that you might have been wrong.
It turns out that the "well-done" was not the case here, but I find it hard to think your post was in good faith.
The conclusion is a bit at odds with the rest of the piece:
> And that’s interesting. When stupid people make a mistake, that’s no big deal. But when brilliant people make a mistake, it’s worth noting.
Maybe not. In the words of the author,
> maybe these are real effects being discovered, but you should at least consider the possibility that you’re chasing noise. When a striking result appears in the dataset, it’s possible that this result does not represent an enduring truth or even a pattern in the general population but rather is just an artifact of a particular small and noisy dataset.
Clearly brilliant people saying that something that later turns out to be noise is statistically undeniable signal is worth noting. I'm not sure what you're seeing here.
I don't think it is worth noting. Brilliant people make mistakes too, just not at the same rate as the rest of us. That one or even a few make the a specific mistake means nothing -- it's statistical fluke, even if it seems noteworthy in isolation.
Kahneman and his book „Thinking fast and slow“ are totaly overrated. In fact „despite Kahneman's previous contributions to the field of decision making, most of the book's ideas are based on 'scientific literature with shaky foundations'. A general lack of replication in the empirical studies cited in the book was given as a justification.“ Unfortunately you will find that statement only at the end of https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow. It should be stated at the beginning. So many wrong assumptions especially in economics, but also management theory are based on this book. But nothing of is book can be replicated: https://replicationindex.com/2020/12/30/a-meta-scientific-pe... Unfortunately far to few people know about this.
I doubt we can get places if the reader of this comment ws to produce thoughts about what I think about what the author thinks psychologists think.
I'm not comfortable with the idea the unmatched creative engineering powers of mother nature did not discover something as simple as radio.
We could power down the electrical grid and do the experiment? But I can just call everyone, what use is telepathy?
The cat is in front of the house when someone arrives (which is never at the same time sometimes skipping a day) but the rest of the day he is nowhere near on the camera footage. For him there is a point, it means food!
Readit News
https://replicationindex.com/2017/02/02/reconstruction-of-a-...
Is it? That's the very basis for error that begat this article. In fact, Turing's and Kahneman's minds were too open. They didn't express sufficient reservation by demanding more rigorous tests of those claims.
Perhaps a better maxim would be, "It's good to have a mind that's open just enough to entertain the impossible."
This article mentions Walter M. Kotschnig as the author of the earliest known instance of the saying, but notes that even Kotschnig seemed to be quoting something that was already known at the time:
https://www.skeptic.com/insight/open-mind-brains-fall-out-ma...
"The problem with having an open mind is that people will insist on coming along and trying to put things in it."
What you are doing here is that you are betraying two huge hidden assumptions that you implicitly make. First is that you are most likely expecting to obtain “mechanism action” in some kind of mathematical terms, which implicitly assumes ability to use mathematics to explain the observed universe. Second is that you are looking for “mechanism of action” for ESP in terms of known physical forces and theories. Here, you implicitly assume that known physics is all there is.
Now, I do not believe in ESP either, my point is only that you are demanding rigor from ESP you don’t from electromagnetism.
Deleted Comment
Deleted Comment
> People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psvchological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions.
(emphasis mine)
Not only does the statement say that “people” make this mistake then go on to cite a questionnaire only of professional psychologists, but the questionnaire is the prototypical non-random random sample the very psychologists taking it are allegedly proving misinformation about. Questionnaires select for people with the time, inclination, and attention to start and then finish a questionnaire.
Infamously, experiments held at universities bias towards undergrads. Experiments that reward participation bias towards people motivated by the reward. Experiments that don’t reward participation bias towards people good-natured enough to contribute just for science.
Randomness is hard.
E.g. if I said "I don't believe in the Holy Book because in verse 7 it says that one cannot trust anything written in any books". Isn't that an analogous reasoning?
I think these examples/arguments are ultimately about exposing a liar-paradox statement, and when you can show such a statement you have proven that something isn't right.
Yes. And might I say everything has a statistical bias. Picking people for an experiment is not like throwing dice
Your sample will always be biased, and it would be crazy to try and get a random sample from "the world". (and even then you have temporal bias - you can't run your experiments on people from the XIX century)
> The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. More important, you must accept that they are true about you.
So so arrogant, and he ended up being wrong about it too. It's very hard to take him seriously after reading this.
Yes. It is. But as another poster has noted, he largely recanted in 2017 [1]. And this is to his credit. Few scholars -- and fewer prominent ones -- have been willing to reverse course in as public a fashion as this.
[1] https://replicationindex.com/2017/02/02/reconstruction-of-a-...
This is a great insight.
So he admits that the studies don't replicate but he still believes in their conclusions? Somehow that makes him look worse, not better, in my eyes but maybe I'm too critical.
Admittedly, the experiments he conducted and cited were found to be flawed. But I think his point, as I understand it, still stands.
Blind faith in the results of a study is not good science. If a study is counter intuitive, one should question it.
Restated: "I'm correct, about us all being prone to cognitive errors, where you like it or not."
> My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young people on the job market. Being associated with a controversial and suspicious field will put them at a severe disadvantage in the competition for positions. Because of the high visibility of the issue, you may already expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the replicability of findings: subliminal perception and dissonance reduction.
> I believe that you should collectively do something about this mess. To deal effectively with the doubts you should acknowledge their existence and confront them straight on, because a posture of defiant denial is selfdefeating. Specifically, I believe that you should have an association, with a board that might include prominent social psychologists from other field. The first mission of the board would be to organize an effort to examine the replicability of priming results, following a protocol that avoids the questions that have been raised and guarantees credibility among colleagues outside the field.
His more recent (2022) retrospective on the episode is also pretty interesting: https://www.edge.org/adversarial-collaboration-daniel-kahnem...
Deleted Comment
Yeah, your reaction is. The whole point was to be humble and accept that you might be wrong. In particular, if well-done research contradicts your prior belief, then you "have no choice" but to consider that you might have been wrong.
It turns out that the "well-done" was not the case here, but I find it hard to think your post was in good faith.
> And that’s interesting. When stupid people make a mistake, that’s no big deal. But when brilliant people make a mistake, it’s worth noting.
Maybe not. In the words of the author,
> maybe these are real effects being discovered, but you should at least consider the possibility that you’re chasing noise. When a striking result appears in the dataset, it’s possible that this result does not represent an enduring truth or even a pattern in the general population but rather is just an artifact of a particular small and noisy dataset.
I'm not comfortable with the idea the unmatched creative engineering powers of mother nature did not discover something as simple as radio.
We could power down the electrical grid and do the experiment? But I can just call everyone, what use is telepathy?
The cat is in front of the house when someone arrives (which is never at the same time sometimes skipping a day) but the rest of the day he is nowhere near on the camera footage. For him there is a point, it means food!
Sheldrake would love it.