While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1].
One would think this is cognitively dissonant enough, but it gets worse:
This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data. On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning. (There's some, yes, but the main thrust of why I believe it is that variants of the experiment have been replicated reliably.)
I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time.
----
Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.
Implicit in all of this is the is-ought problem.[0] The data are collected and interpreted under some procedure, often with normative biases built in about how the world ought to be (especially when involving human subjects), but are interpreted as saying what the world is. Thus data collection is fertile ground for charlatans.
When the psychiatric profession or Google or whoever else use experimentation to decide on what criteria they should follow, with sound controls, valid statistical analysis and loads of replication, they either arrive at evaluation procedures without much bias or, more likely, they realize the phenomenon they're trying to measure is almost all noise with no or excessively weak signals.
A better approach would be to acknowledge as much normative bias as possible up front, then conduct tests using sound experimental design. But the problem with this approach is that the data shows performing a bunch of well-crafted experiments is expensive, and management doesn't buy in if the vast majority are unlikely to reject the null. That leaves us which a class of "data driven" managers who are in fact indulging their biases to a sometimes extreme degree, using "the data" as a shield.
I find it strange that these are presented in tension, when they’re complementary.
You can create situations where you have a lot of data but can’t reach conclusions, because you lack a narrative and explanatory model which “makes sense” of that data; inversely, you can convincingly argue complete nonsense that’s obviously contrary to facts.
Deep understanding requires a model/narrative which fits the collection of data we have, and which allows us to reason about and predict the outcome of new situations.
As Jeff Bezos put it:
> Good inventors and designers deeply understand their customer. They spend tremendous energy developing that intuition. They study and understand many anecdotes rather than only the averages you’ll find on surveys. They live with the design.
> I’m not against beta testing or surveys. But you, the product or service owner, must understand the customer, have a vision, and love the offering. Then, beta testing and research can help you find your blind spots. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste. You won’t find any of it in a survey.
I was about to write that in case of Bezos with Amazon, the customer was simpler and the answer was to just pour money into it until you substituted the market, but I realise now that that is not that simple. It seems simple because we have hindsight.
My main idea though is that it is very hard to foresee what the customer will want after you deliver the product. Not what the customers want now, because sometimes they don't understand it until they experience it, and that makes me think that there is a LOT of luck at play here and a good deal of continency in prototype product design. Experience alone could be overrated. Think Kodak, I don't think they didn't have experience in product design, that they didn't understand their customers. I think they only didn't risk their luck and didn't think about what their customers would want in the future. And that is always a gamble.
- Things are more nuanced and complex than I am putting it here, but bottom line is that I am trying to tap into survivors bias.
Seems far-fetched to assume that this thesis applies to product development just the same?
The impact a data-driven mindset can have on the organization cannot be understated ('RIP intrinsic motivation' section). I've seen it first-hand, both data being used as cop-out for bad leadership, meaningless 'successes' used as trading cards for promotions, and design experts having a decade of experience overridden by shaky statistical analysis, or worse, non-inferiority tests.
Meanwhile, the shortcomings in the product that everyone knows are rarely addressed because they are 'difficult to test'.
> They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment.
I came across this in Thinking Fast and Slow. Kahneman was a big fan of Meehl and restates the point:
The important conclusion from this research is that an algorithm that is constructed on the back of an envelope is often good enough to compete with an optimally weighted formula, and certainly good enough to outdo expert judgment.
I too agree with the premise of this article. On this topic of expert judgment vs data, however, I found the counterpoint in this HN comment thought-provoking enough to bookmark and refer back to now and again:
I started at MS during Vista and I've been involved (sometimes tangentially) with Windows ever since. This is all my opinion, but It's been very interesting seeing the decision making process change over time.
If I had to summarize the change, I'd say that it's evolved from an expertise-based system to a data based system. The reason why eight people were present at every planning meeting is because their expert opinion was the primary tool used in decision making. In addition to poor decisions, this had two very negative outcomes:
1) reputation was fiercely fought for. Individuals feared that if they were ever incorrect, the damage to their reputation would limit their ability to impact future decisions and eventually lead to career death. Whether this actually happened or not is irrelevant; the fear itself caused overt caution and consensus seeking.
2) In the absence of data, an eloquent negotiator is often able to obtain their desired outcome, no matter how sub-optimal that outcome might be.
It seems to imply that expertise-driven design gave us Vista and Win7 while the data-driven one gave us Win8, Win10, and Win11. It's notable that, from this list, Win7 seems to be the only one that people genuinely liked.
> Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.
Highlighting your edit at the bottom, as I think it’s important and not everyone will read that far.
I've come to heavily discount these types of studies. What makes an expert? What was the sample size of experts? What was the non-expert tool? Etc.
There is such a thing as having common sense based on thoughtful life experience. Checklists and regressions help, but human beings are very capable of deep expertise and to pretend otherwise is silly. I expect a musician to be able to identify a violin from a viola.
>Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.
Maybe too much of a nit-pick, but how does one build expertise without data? I'll grant that it may be informally or subconsciously collected but it's still data.
It makes me think of Malcolm Gladwell's book Blink. There are lots of experts who can subconsciously chunk data to make intuitive and reliable decisions. But they got to that point often gathering lots of data in the form of experience.
> This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data.
I'm not sure what you're claiming. All intellectual demonstration is a matter of rational argument. That's what proofs are: arguments. Data is not self-explanatory or demonstration. "Data" can only support arguments by first being collected, something motivated by argument, and then interpreted so that it can enter into argument as a body of propositions.
> On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning.
I don't understand. Argument is logical demonstration. The strongest form is the deductive argument. If you don't have a logical argument, then you haven't got a demonstration.
> I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time.
Depending on what you mean, this could be good or bad. Inconsistency is not a virtue, and if there is an inconsistency between two of your beliefs, then it means you've got work to do (or at least you'll need to admit you don't know what the truth is). This requires humility, the frank acknowledgment that you're faced with an aporia that you don't know (at least not yet) how to address. It also requires patience if you are to tolerate your ignorance instead of jumping to some ersatz explanation.
I feel like the author is leaning into comfort, intuitiveness. You bring up a fantastic point. Often we find data reveals things very unintuitive to human experience. We should always try to make Good Arguments - but without data they aren't always honest beyond feelings.
> Are you prepared to do some very very fancy statistics?
I'd extend this with "... while understanding what you're doing?"
I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.
I think "Be brutally honest about you many assumptions and caveats" at least implies that.
I mean, in an informal setting there's room for an honest person to say "well I did some math and I don't really get it but I think it says...," but I think this article is addressed to software engineers and scientists. Someone representing themself as an engineer or scientists has a professional ethical responsibility to some sort of... I dunno, epistemic honesty, the knowledge of what their expertise covers, and communicating their limitations to laymen.
The person with the A/B test in your example is either a liar because they are misrepresenting what their tool says, or they are a liar because they are misrepresenting their ability to tell you what it says, but either way they are a liar.
I have experienced this first hand, so this article resonates a lot with me.
I worked with a manager who prioritized work which was easily measurable, so he could report the good numbers to leadership and get career points out of this. Unfortunately the project we took on was a demanding and technically challenging problem, and in almost a year of work of a team of engineers we made barely any real progress or made any actual difference, but the numbers were great and people were satisfied during presentations. I ended up feeling completely disconnected from my job and losing all motivation to work there.
> I originally claimed that data-driven culture leads bad arguments involving data to be favored over good arguments that don’t
This is symptomatic of the deeper problem of thinking in terms of bumper stickers and slogans, instead of thinking from first principles. When it afflicts educated people, usually you hear slogans like "an anecdote is not data", or "that's the slippery slope fallacy". Instead of grappling with noisy reality, they have sharp cognitive categories with firm boundaries between concepts, then they try to squeeze things into these categories in order to make cognition easier because the relations between the categories are already understood. This gives them the illusion of rigorous and clear thought.
This entire discussion makes a good case for why the general populace would benefit from being taught the basics of philosophy.
In this case the topic of value is the often fraught relationship between empiricism and rationalism, and the impacts each have on the scientific process, research, education, and how we go about understanding the world.
To operate with one with a complete absence of the other is to expose yourself to huge, often fundamental gaps in your thinking, your arguments, and your plans. This is what the author is ultimately getting at from the direction of the empirical: data, in the form of a large collection of discrete observations, can be used to justify a sea of mutually exclusive claims that may or may not be in accordance with reality, and that's to say nothing about the quality of the data itself.
Most argumentation we do as on questions that are worth debating aren't based purely on deductive reasoning, but more on informal reasoning and heuristics with limited evidence.
Toulmin identifies the three essential parts of any argument as
- the claim
- the data (also called grounds or evidence), which support the claim
- the warrant.
The warrant is the assumption on which the claim and the evidence depend. Another way of saying this
would be that the warrant explains why the data support the claim.
Toulmin says that the weakest part of any argument is its weakest warrant. Remember that the warrant is
the link between the data and the claim. If the warrant isn’t valid, the argument
collapses.
Example:
Claim: You should buy our toothwhitening product.
Data or Grounds: Studies show that teeth are 50% whiter after using the product for a specified time.
Warrant: People want whiter teeth.
Notice that those commercials don’t usually bother trying to convince you that you want whiter teeth;
instead, they assume that you have accepted the value our culture places on whiter teeth.
From there you can go into the whole spectrum of critical thinking approaches, and then on to what's basically the liberal arts e.g. philosophy, social sciences etc. as you desire. But the value you get from all of those things depends heavily on the framework you have for thinking about them going in.
Claiming random things are "fake news" would be a lot harder if people could work out what is and isn't fake by themselves!
> I would start by simply putting everyone through a course in deductive reasoning at the earliest age possible
I was taught the explicit premise of deductive vs. inductive reasoning as part of our unit on the scientific method in, I think, fourth or fifth grade. I always assumed this was a standard curriculum module.
>I would start by simply putting everyone through a course in deductive reasoning at the earliest age possible
Indeed. This would help ensure people's brains' transition function is stable enough to perform faultless computation.
We forget that our brains aren't wired for exact computation. They're wired to perform approximations of computation that are good enough for survival.
As a result, you end up with myriads students who go through the school system via memorization and emergent fuzzy computation.
They reach an adult age without possessing the cognitive tool-set to grasp the subtleties and nuances of the world they live in.
The fact that such people are also preyed on by charlatans, ad companies and politicians(intersection of charlatans and ad companies) obviously doesn't help.
The point I would add is that hardly anyone uses the empirical process directly. It is all 'this article claims this' or 'this study says that'. It's very 'meta' with little to no personal verification or testing of the claims - ie, theories based on theories or models based on models, or maps based on maps.
Very few check the terrain itself to confirm that the map applies. We trust education, experts, peer review etc. We're drowning in models, especially as these are easily represented on computers, but have no ability to check the models against reality.
PS this disassociation from reality will not improve as we move forward technologically. No doubt, in the metaverse we will be able to create ever more elaborate models, or is it that we will be ever more disassociated from our own anecdotal experiences? (Where 'anecdotal' is something to apologise about).
> This entire discussion makes a good case for why the general populace would benefit from being taught the basics of philosophy.
But our entire education pipeline is optimized for loading people into the “system”. Philosophy etc. has little market value (unless it aligns with the system).
While this is indeed an issue that falls within the domain of philosophy, philosophy is also home to fields in which the absence of empirical evidence is regarded as an irrelevance, or at best a mere detail that can be deferred to an indefinite future. Take, for example, the resurgence of enthusiasm for panpsychism, and the enduring appeal of armchair metaphysics.
I am doubtful that academic philosophy has much enthusiasm for pursuing and inculcating the practical aspects of reason (any more than does theoretical physics or mathematics), though there are exceptions.
Exploring ideas like panpsychism doesn't mean you're committing to them being true. We can't know everything, and we can't always link new ideas deductively to things we are certain about, but we can notice the inadequacy of current explanations, say "suppose this explanation is true" and proceed from there. Every good philosopher knows that they're doing that. And the fact that people defend their position and attack opposing views is just part of the adversarial process for testing ideas. Yeah, of course ego and pride and hubris happen to many philosophers, and the academic profession is frankly in a bad state, but that doesn't mean the fundamental approach is bad.
Idk how if studying philosophy helps. Most philosophers were/are themselves committed to one school or theory, with gaps galore.
In any case, I think empirical science's defeat of rationalism ( eg Galileo Vs Church) has all sorry of ramifications. Social sciences like economics and psychology have a lot of trouble bridging the gaps.
The problem with this is that philosophy isn't a magical panacea that illuminates the way towards a more ideal state. It can be used to justify a sea of mutually exclusive claims that may not be in accordance with reality, and that's to say nothing about the quality of the arguments themselves.
I often experience the inverse: people come up with hypotheses and theories that should see expressions in observable data - but no-one bothers to look and instead everyone argues around logical constructs etc.
This reminds me a lot of the discussion of the scientific method by Karl Popper, and David Deutsch who was very influenced by Popper. "Being data-driven" sounds very empirical. Just look at the data, and see what you find in it.
But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data. Popper in Conjectures and Refutations:
> Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems.
Deutsch, in The Beginning of Infinity, emphasizes the importance of conjecture, and the role of observation as refuting or criticising those conjectures:
> Where does [knowledge] come from? Empiricism said that we derive it from sensory experience. This is false. The real source of our theories is conjecture, and the real source of our knowledge is conjecture alternating with criticism. We create theories by rearranging, combining, altering and adding to existing ideas with the intention of improving upon them. The role of experiment and observation is to choose between existing theories, not to be the source of new ones. We interpret experiences through explanatory theories, but true explanations are not obvious.
To bring this back to the subject of the article, I might suggest that it's possible to be "data driven" without a sound explanation or theory that the data is either interpreted through, or used to criticise. Or maybe such theories do exist, but are left implicit.
Doesn’t the scientific method specifically say you can’t start with the data, you have to start with a hypothesis otherwise you are subject to all sorts of selection/hindsight biases. I mean you can start with data, but then you have to develop a hypothesis and use that to create an experiment that generates new data in order to reach a conclusion. It seems like that is the compromise the author is looking for, start with a good idea, then see if you can verify it with data.
The scientific method as taught in K–12 schools is largely pablum. Often, the real process (beyond iterating off prior research) begins with collecting data, then noticing patterns to make a hypothesis to be tested with targeted data collection.
A single metric is just one very thin dimension from the temporal development of a complex process involving many factors. You need to watch a multitude of metrics to devise an explanative theory, and even then, that theory can be rendered flawed when new and unexpected factors come at play.
The fact that empiricism is false was a revelation to me as a young adult, after reading so much about the triumphs of science and reason and getting very excited (mistakenly) that you can get away without bothering with pesky things like epistemology. Of course, Quine and others pointed out that empirical observations are useless without explanations both of the phenomenon being measured and the measurement device itself (including, for example, the human vision system). And I believe it was Deutschmark who pointed out that empiricism is itself an epistemology which had to be invented. It turns out that it tended to be a significant improvement upon previous widespread epistemologies, but that doesn’t mean it’s not false. :)
I agree, pure empiricism can lead to superstition. If you only learn from experience, and do not have any theory that ensures the consistency of the model, it's easy to infer wrong causal connections.
> Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems.
Thanks, I'd never heard this quote before. He's pretty much describing pragmatism à la William James. I had no idea.
The pragmatists went a little bit too far in my opinion, though it has been a long time since I read any of them. Popper is describing observations, not reality.
I highly recommend Conjectures if you can find a copy. It's a short read and interesting.
I won’t belabor the point because others have already made it: this article assumes there is some way to sort through good and bad arguments in the absence of data - a pretty big leap. The reality is all of our arguments are appealing to some sort of data (eg previous experience), it’s just that it doesn’t always fit in a neat definition of data.
"Previous experience" is not what is meant by 'data' in this industry. If company's decision-making was including both data and experience/wisdom/intuition, it wouldn't be so frustratingly wrong all the time.
I agree that's not what is meant by 'data' in the industry and I'm challenging that a little bit. However, even if we use the industry definition, what you're saying is hyperbole. Every company uses both data and experience to varying degrees. People get hung up when they think the balance isn't appropriate - not surprisingly, that happens when one or the other doesn't support their opinion. I'd rather be in a position of defending my opinion with data. It's already been quoted but... "If we have data, let's look at data. If all we have are opinions, let's go with mine."
Readit News
One would think this is cognitively dissonant enough, but it gets worse:
This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data. On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning. (There's some, yes, but the main thrust of why I believe it is that variants of the experiment have been replicated reliably.)
I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time.
----
Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.
----
[1]: Here's just one relatively early example: http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/...
When the psychiatric profession or Google or whoever else use experimentation to decide on what criteria they should follow, with sound controls, valid statistical analysis and loads of replication, they either arrive at evaluation procedures without much bias or, more likely, they realize the phenomenon they're trying to measure is almost all noise with no or excessively weak signals.
A better approach would be to acknowledge as much normative bias as possible up front, then conduct tests using sound experimental design. But the problem with this approach is that the data shows performing a bunch of well-crafted experiments is expensive, and management doesn't buy in if the vast majority are unlikely to reject the null. That leaves us which a class of "data driven" managers who are in fact indulging their biases to a sometimes extreme degree, using "the data" as a shield.
[0]https://plato.stanford.edu/entries/hume-moral/#io
You can create situations where you have a lot of data but can’t reach conclusions, because you lack a narrative and explanatory model which “makes sense” of that data; inversely, you can convincingly argue complete nonsense that’s obviously contrary to facts.
Deep understanding requires a model/narrative which fits the collection of data we have, and which allows us to reason about and predict the outcome of new situations.
As Jeff Bezos put it:
> Good inventors and designers deeply understand their customer. They spend tremendous energy developing that intuition. They study and understand many anecdotes rather than only the averages you’ll find on surveys. They live with the design.
> I’m not against beta testing or surveys. But you, the product or service owner, must understand the customer, have a vision, and love the offering. Then, beta testing and research can help you find your blind spots. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste. You won’t find any of it in a survey.
https://www.aboutamazon.com/news/company-news/2016-letter-to...
My main idea though is that it is very hard to foresee what the customer will want after you deliver the product. Not what the customers want now, because sometimes they don't understand it until they experience it, and that makes me think that there is a LOT of luck at play here and a good deal of continency in prototype product design. Experience alone could be overrated. Think Kodak, I don't think they didn't have experience in product design, that they didn't understand their customers. I think they only didn't risk their luck and didn't think about what their customers would want in the future. And that is always a gamble.
- Things are more nuanced and complex than I am putting it here, but bottom line is that I am trying to tap into survivors bias.
The impact a data-driven mindset can have on the organization cannot be understated ('RIP intrinsic motivation' section). I've seen it first-hand, both data being used as cop-out for bad leadership, meaningless 'successes' used as trading cards for promotions, and design experts having a decade of experience overridden by shaky statistical analysis, or worse, non-inferiority tests.
Meanwhile, the shortcomings in the product that everyone knows are rarely addressed because they are 'difficult to test'.
I came across this in Thinking Fast and Slow. Kahneman was a big fan of Meehl and restates the point:
The important conclusion from this research is that an algorithm that is constructed on the back of an envelope is often good enough to compete with an optimally weighted formula, and certainly good enough to outdo expert judgment.
https://www.goodreads.com/quotes/9574537-the-important-concl...
I too agree with the premise of this article. On this topic of expert judgment vs data, however, I found the counterpoint in this HN comment thought-provoking enough to bookmark and refer back to now and again:
I started at MS during Vista and I've been involved (sometimes tangentially) with Windows ever since. This is all my opinion, but It's been very interesting seeing the decision making process change over time.
If I had to summarize the change, I'd say that it's evolved from an expertise-based system to a data based system. The reason why eight people were present at every planning meeting is because their expert opinion was the primary tool used in decision making. In addition to poor decisions, this had two very negative outcomes:
1) reputation was fiercely fought for. Individuals feared that if they were ever incorrect, the damage to their reputation would limit their ability to impact future decisions and eventually lead to career death. Whether this actually happened or not is irrelevant; the fear itself caused overt caution and consensus seeking.
2) In the absence of data, an eloquent negotiator is often able to obtain their desired outcome, no matter how sub-optimal that outcome might be.
https://news.ycombinator.com/item?id=15174737#15176957
Even more provocative, it ends up being a (qualified, as I read it) defense of telemetry.
Highlighting your edit at the bottom, as I think it’s important and not everyone will read that far.
There is such a thing as having common sense based on thoughtful life experience. Checklists and regressions help, but human beings are very capable of deep expertise and to pretend otherwise is silly. I expect a musician to be able to identify a violin from a viola.
Maybe too much of a nit-pick, but how does one build expertise without data? I'll grant that it may be informally or subconsciously collected but it's still data.
It makes me think of Malcolm Gladwell's book Blink. There are lots of experts who can subconsciously chunk data to make intuitive and reliable decisions. But they got to that point often gathering lots of data in the form of experience.
I'm not sure what you're claiming. All intellectual demonstration is a matter of rational argument. That's what proofs are: arguments. Data is not self-explanatory or demonstration. "Data" can only support arguments by first being collected, something motivated by argument, and then interpreted so that it can enter into argument as a body of propositions.
> On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning.
I don't understand. Argument is logical demonstration. The strongest form is the deductive argument. If you don't have a logical argument, then you haven't got a demonstration.
> I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time.
Depending on what you mean, this could be good or bad. Inconsistency is not a virtue, and if there is an inconsistency between two of your beliefs, then it means you've got work to do (or at least you'll need to admit you don't know what the truth is). This requires humility, the frank acknowledgment that you're faced with an aporia that you don't know (at least not yet) how to address. It also requires patience if you are to tolerate your ignorance instead of jumping to some ersatz explanation.
I'd extend this with "... while understanding what you're doing?"
I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.
I mean, in an informal setting there's room for an honest person to say "well I did some math and I don't really get it but I think it says...," but I think this article is addressed to software engineers and scientists. Someone representing themself as an engineer or scientists has a professional ethical responsibility to some sort of... I dunno, epistemic honesty, the knowledge of what their expertise covers, and communicating their limitations to laymen.
The person with the A/B test in your example is either a liar because they are misrepresenting what their tool says, or they are a liar because they are misrepresenting their ability to tell you what it says, but either way they are a liar.
IF you need 'fancy' statistics then it is not going to be a good data driven argument at all.
I worked with a manager who prioritized work which was easily measurable, so he could report the good numbers to leadership and get career points out of this. Unfortunately the project we took on was a demanding and technically challenging problem, and in almost a year of work of a team of engineers we made barely any real progress or made any actual difference, but the numbers were great and people were satisfied during presentations. I ended up feeling completely disconnected from my job and losing all motivation to work there.
This is symptomatic of the deeper problem of thinking in terms of bumper stickers and slogans, instead of thinking from first principles. When it afflicts educated people, usually you hear slogans like "an anecdote is not data", or "that's the slippery slope fallacy". Instead of grappling with noisy reality, they have sharp cognitive categories with firm boundaries between concepts, then they try to squeeze things into these categories in order to make cognition easier because the relations between the categories are already understood. This gives them the illusion of rigorous and clear thought.
In this case the topic of value is the often fraught relationship between empiricism and rationalism, and the impacts each have on the scientific process, research, education, and how we go about understanding the world.
To operate with one with a complete absence of the other is to expose yourself to huge, often fundamental gaps in your thinking, your arguments, and your plans. This is what the author is ultimately getting at from the direction of the empirical: data, in the form of a large collection of discrete observations, can be used to justify a sea of mutually exclusive claims that may or may not be in accordance with reality, and that's to say nothing about the quality of the data itself.
Toulmin identifies the three essential parts of any argument as
The warrant is the assumption on which the claim and the evidence depend. Another way of saying this would be that the warrant explains why the data support the claim.Toulmin says that the weakest part of any argument is its weakest warrant. Remember that the warrant is the link between the data and the claim. If the warrant isn’t valid, the argument collapses.
Example:
Notice that those commercials don’t usually bother trying to convince you that you want whiter teeth; instead, they assume that you have accepted the value our culture places on whiter teeth.https://www.blinn.edu/writing-centers/pdfs/Toulmin-Argument....
From there you can go into the whole spectrum of critical thinking approaches, and then on to what's basically the liberal arts e.g. philosophy, social sciences etc. as you desire. But the value you get from all of those things depends heavily on the framework you have for thinking about them going in.
Claiming random things are "fake news" would be a lot harder if people could work out what is and isn't fake by themselves!
I was taught the explicit premise of deductive vs. inductive reasoning as part of our unit on the scientific method in, I think, fourth or fifth grade. I always assumed this was a standard curriculum module.
Indeed. This would help ensure people's brains' transition function is stable enough to perform faultless computation. We forget that our brains aren't wired for exact computation. They're wired to perform approximations of computation that are good enough for survival.
As a result, you end up with myriads students who go through the school system via memorization and emergent fuzzy computation.
They reach an adult age without possessing the cognitive tool-set to grasp the subtleties and nuances of the world they live in. The fact that such people are also preyed on by charlatans, ad companies and politicians(intersection of charlatans and ad companies) obviously doesn't help.
The point I would add is that hardly anyone uses the empirical process directly. It is all 'this article claims this' or 'this study says that'. It's very 'meta' with little to no personal verification or testing of the claims - ie, theories based on theories or models based on models, or maps based on maps.
Very few check the terrain itself to confirm that the map applies. We trust education, experts, peer review etc. We're drowning in models, especially as these are easily represented on computers, but have no ability to check the models against reality.
PS this disassociation from reality will not improve as we move forward technologically. No doubt, in the metaverse we will be able to create ever more elaborate models, or is it that we will be ever more disassociated from our own anecdotal experiences? (Where 'anecdotal' is something to apologise about).
But our entire education pipeline is optimized for loading people into the “system”. Philosophy etc. has little market value (unless it aligns with the system).
Deleted Comment
I am doubtful that academic philosophy has much enthusiasm for pursuing and inculcating the practical aspects of reason (any more than does theoretical physics or mathematics), though there are exceptions.
In any case, I think empirical science's defeat of rationalism ( eg Galileo Vs Church) has all sorry of ramifications. Social sciences like economics and psychology have a lot of trouble bridging the gaps.
> Most philosophers were/are themselves committed to one school or theory, with gaps galore.
Most scientists specialize one thing, but students of science don't. One can learn about many schools of philosophy, as well.
But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data. Popper in Conjectures and Refutations:
> Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems.
Deutsch, in The Beginning of Infinity, emphasizes the importance of conjecture, and the role of observation as refuting or criticising those conjectures:
> Where does [knowledge] come from? Empiricism said that we derive it from sensory experience. This is false. The real source of our theories is conjecture, and the real source of our knowledge is conjecture alternating with criticism. We create theories by rearranging, combining, altering and adding to existing ideas with the intention of improving upon them. The role of experiment and observation is to choose between existing theories, not to be the source of new ones. We interpret experiences through explanatory theories, but true explanations are not obvious.
To bring this back to the subject of the article, I might suggest that it's possible to be "data driven" without a sound explanation or theory that the data is either interpreted through, or used to criticise. Or maybe such theories do exist, but are left implicit.
If you look at the heart attack data, and you ignore smoking you end up inventing the mythical Type A personality — but it was data driven.
https://en.m.wikipedia.org/wiki/Type_A_and_Type_B_personalit...
Thanks, I'd never heard this quote before. He's pretty much describing pragmatism à la William James. I had no idea.
I highly recommend Conjectures if you can find a copy. It's a short read and interesting.
Here is a good talk https://www.youtube.com/watch?v=EVwjofV5TgU
Obligatory: https://en.m.wikipedia.org/wiki/All_models_are_wrong