> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.)
This coupled with people acting on its predictions is a kind of self fulfilling prophecy.
which is to ask, are AI safety folks building models of this pattern? :)
This is true for a lot of things, not just AI. But in AI, a guy who didn't get a High School degree and wrote Harry Potter Fan Fiction is one of the leading voices in doomerism.
The problem is you can't just "use logic and reason" because simple models are not good enough. The nuance dominates, but that's why we have experts.
What's funny to me is that people will confidently argue with experts and others value their opinion over the expert's knowledge. But on the other hand, people tend to just take machines at face value. Maybe these aren't overlapping groups, but it does appear that way. There's a great irony in trusting a machine but not the person/s that built said machine.
I don’t trust the machines or the people who make them, and I didn’t have to read the Harry Potter fanfic to know ad hominems are poor arguments. What group does that make me?
Not sure if you intended this, but it feels like the first sentence of your argument is more broadly a critique of the credentials of AI Safety proponents. Maybe you are distinguishing between doomers vs broader AI Safety proponents, but if not, I feel like the counterargument is that most people on the CAIS letter (https://www.safe.ai/work/statement-on-ai-risk) interface quite frequently with these AI models and are also (purportedly) seriously concerned about AI safety
> are AI safety folks building models of this pattern?
First you need to ask if AI "safety folks" actually understand the technology, and if they are thinking about it objectively. If they believe that we're a few years away from accidentally creating Skynet, they need to put down the crack pipe and go work in another field.
If you knew everyone would ask gpt before doing anything, you would make gpt say what woudl generally be considered the better option. Not going to war, not committing suicide, etc. In this way even if war was the optimal decision according to some other utility function, the behavior of people is directed in a positive way. (Presumably)
"Facially" in the sense of "on the face of it", roughly as a synonym for "obviously", seems like a pretty standard usage to me—this is certainly not the first place I've seen the word used in this sense.
Human recall failure. Probably wanted "seemingly", "apparently", or even "ostensibly", but who's got time for all that when the publish button's right there.
> Also, ML code tends to vastly more complex and less standardized than traditional statistical modeling.
I mean, hey, it's proof that the text isn't AI generated, since ChatGPT is better at English than that, but it makes it hard to read and I'm not going to buy their book if it's going to be full of errors like that.
If this is already such a problem even in the professional discipline and vocation whose sine qua non is the accurate analysis of physical reality, I'm really nervous about the next few years. And I was nervous already...
In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)
> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)
It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstraction.
Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task.
I asked it "what is the PS3 game where the full version of To Kill a Mockingbird is in there?" and it responded back with "The Sabateour", when the correct answer would have been "The Darkness". That is incorrect by most definitions of the word, whether or not it's a consequence of the training model doesn't really change that.
I suppose we could get into details about epistemology and ontology about the nature of what an answer "is", but I think it's fair to say that "incorrect" is when it gives you something that doesn't accomplish the task you asked it to do, or rather when it tries to accomplish the task but what it gives you don't work.
This is like saying the arguments put forth by a schizophrenic lawyer are rational and correct.
If the context is that it's a tool, correct is defined as reality within the context of the use of that tool. If it's to find facts, it can be incorrect, since the context of a fact is reality. If it's writing a story, then "correct" would be based on continuity, etc.
If you're using it as a tool to generate words related to previous ones, then sure, it's always correct, but that's not probably not a useful tool for most people. But, being a next word predictor doesn't mean it can't also be a useful tool in real world contexts. There are, literally, billions of dollars being spent on pushing them to be more "correct" in more contexts, so it's a useful concept being considered, even though they're "just" next word predictors.
While yes, this is the technical reason — it’s important to not overlook how non-technical people see LLMs. And not only that, how they are being marketed.
I’m struggling to think of any comparable technology where the regular median users understanding is both fundamentally wrong— and is being purposefully misinformed.
“Correctness” is a property of a proposition determined by an observer. Sometimes what is output by an LLM is correct, sometimes not. That an LLM is aware of the output or not means literally nothing.
This habit of latching onto one word specifically to ignore what everyone knows is obnoxious, pedantic, and most of the time not even technically correct. It's just stupid quibbling over how words in English can be used to mean different things. And just so you know, the model doesn't "learn" anything, you're just adjusting weights until you get a desired result.
People treating tools like they're infallible has been a problem since computers were invented, but IMHO the biggest difference with AI is how confident and convincing it can be in its output. Much like others here, I already have had to convince, very carefully, many otherwise-decently-intelligent people who believed ChatGPT was correct.
Thus I think the biggest success of AI will be the arts, where imprecision is not fatal, and hallucinations turn into entertainment instead of "truths".
I think this misses something important. If it makes economic sense, corporations will figure out ways to integrate AI into their processes, even if it's imperfect. After all, companies are already built out of humans who are also often confidently wrong - but successful companies have ways to detect and mitigate that. In fact, that's one of the primary requirements for a company to survive, that it's able to build a functioning system out of imperfect components, particularly humans.
You can see an example of this in the use of LLMs to generate code. In that case, there's a whole SDLC pipeline designed to detect errors: type systems, language compilers and runtimes, tests of various kinds, QA, user feedback, etc. We don't just trust confident software developers to produce correct code.
Even a life-critical function like medical imaging - where imprecision can be fatal - can potentially benefit from this, where AI is used in conjunction with human review. It mainly requires development of some standards of practice - unlike with an average user blindly trusting the output of a model, radiologists would need training on how to use the models in question.
AI is a tool … a fool with a tool is still a fool … For natural sciences, there is no need to worry since nature would provide the ultimate check … for social “sciences”, it is entirely a different story.
To be fair, people did this before ChatGPT. It's just the thing they point to as evidence now, and they'll always find something. The underlying problem is much bigger:
1) people confidently arguing with domain experts about topics that they have little to no experience in.
2) people valuing the opinions of arguers from 1 over experts.
To be extra fair, "domain experts" in some areas have had a bad few years; there are a couple of fields I can think of off the top of my head where the "experts" wheeled out to advise/scare the public are clearly more influenced by politics (or saving their own skin) than science. Replacing trust in experts with trust in LLMs is obviously dumb, but who is Joe Sixpack supposed to turn to?
Wow I came into this article angry, idk if their book title accurately conveys the sober, expert analysis it contains! In case anyone else is curious why they’re talking about “leakage” in the first place instead of the existing term “model bias”, here’s the paper they cite in the “compelling evidence” paper that started these two’s saga with the snake oil salesmen: https://www.cs.umb.edu/~ding/history/470_670_fall_2011/paper...
Crux passage:
> Our focus here is on leakage, which is a specific form of illegitimacy that is an intrinsic property of the observational inputs of a model. This form of illegitimacy remains partly abstract, but could be further defined as follows: Let u be some random variable. We say a second random variable v is u-legitimate if v is observable to the client for the purpose of inferring u. In this case we write v € legit{u}.
> A fully concrete meaning of legitimacy is built-in to any specific inference problem. The trivial legitimacy rule, going back to the first example of leakage given in Section 1, is that the target itself must never be used for inference:
> (1) y !€ legit{y}
So ultimately this all about bad experimental discipline re: training and test data, in an abstract way? I’ve been staring at this paper for way too long trying to figure out what exactly each “target” is and how it leaks, but I hope that engineering-translation is close
Readit News
This coupled with people acting on its predictions is a kind of self fulfilling prophecy.
which is to ask, are AI safety folks building models of this pattern? :)
The problem is you can't just "use logic and reason" because simple models are not good enough. The nuance dominates, but that's why we have experts.
What's funny to me is that people will confidently argue with experts and others value their opinion over the expert's knowledge. But on the other hand, people tend to just take machines at face value. Maybe these aren't overlapping groups, but it does appear that way. There's a great irony in trusting a machine but not the person/s that built said machine.
First you need to ask if AI "safety folks" actually understand the technology, and if they are thinking about it objectively. If they believe that we're a few years away from accidentally creating Skynet, they need to put down the crack pipe and go work in another field.
"facially impossible" ... does that really riff on "on the face of it", or is it farcically misspelt?
Garbage in, garbage out 8)
> Also, ML code tends to vastly more complex and less standardized than traditional statistical modeling.
I mean, hey, it's proof that the text isn't AI generated, since ChatGPT is better at English than that, but it makes it hard to read and I'm not going to buy their book if it's going to be full of errors like that.
Dead Comment
It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstraction.
I asked it "what is the PS3 game where the full version of To Kill a Mockingbird is in there?" and it responded back with "The Sabateour", when the correct answer would have been "The Darkness". That is incorrect by most definitions of the word, whether or not it's a consequence of the training model doesn't really change that.
I suppose we could get into details about epistemology and ontology about the nature of what an answer "is", but I think it's fair to say that "incorrect" is when it gives you something that doesn't accomplish the task you asked it to do, or rather when it tries to accomplish the task but what it gives you don't work.
If the context is that it's a tool, correct is defined as reality within the context of the use of that tool. If it's to find facts, it can be incorrect, since the context of a fact is reality. If it's writing a story, then "correct" would be based on continuity, etc.
If you're using it as a tool to generate words related to previous ones, then sure, it's always correct, but that's not probably not a useful tool for most people. But, being a next word predictor doesn't mean it can't also be a useful tool in real world contexts. There are, literally, billions of dollars being spent on pushing them to be more "correct" in more contexts, so it's a useful concept being considered, even though they're "just" next word predictors.
I’m struggling to think of any comparable technology where the regular median users understanding is both fundamentally wrong— and is being purposefully misinformed.
Thus I think the biggest success of AI will be the arts, where imprecision is not fatal, and hallucinations turn into entertainment instead of "truths".
You can see an example of this in the use of LLMs to generate code. In that case, there's a whole SDLC pipeline designed to detect errors: type systems, language compilers and runtimes, tests of various kinds, QA, user feedback, etc. We don't just trust confident software developers to produce correct code.
Even a life-critical function like medical imaging - where imprecision can be fatal - can potentially benefit from this, where AI is used in conjunction with human review. It mainly requires development of some standards of practice - unlike with an average user blindly trusting the output of a model, radiologists would need training on how to use the models in question.
1) people confidently arguing with domain experts about topics that they have little to no experience in.
2) people valuing the opinions of arguers from 1 over experts.
There, fixed the title.
I think this is even better
Dead Comment
Crux passage:
> Our focus here is on leakage, which is a specific form of illegitimacy that is an intrinsic property of the observational inputs of a model. This form of illegitimacy remains partly abstract, but could be further defined as follows: Let u be some random variable. We say a second random variable v is u-legitimate if v is observable to the client for the purpose of inferring u. In this case we write v € legit{u}.
> A fully concrete meaning of legitimacy is built-in to any specific inference problem. The trivial legitimacy rule, going back to the first example of leakage given in Section 1, is that the target itself must never be used for inference:
> (1) y !€ legit{y}
So ultimately this all about bad experimental discipline re: training and test data, in an abstract way? I’ve been staring at this paper for way too long trying to figure out what exactly each “target” is and how it leaks, but I hope that engineering-translation is close