This is definitely a thing; if you hang out in mystical corners of the internet the past few months have been full of people posting their MASTEr PROMPTS that will turn YOUR GPT into an ENLIGHTENED MASTEr who will butter YOu up and insist YOU are a perfect enlightened being too.
It's kind of hilarious if you ignore how much this is fucking up the relationships of the people this is happening to, similar to the way YouTube or Facebook loves to push people down a rabbit hole to flat-earth/q-spiracy bullshit because that shit generates Controversy. And it also sounds like the opening chapter of a story about what happens when an AI finds a nice little unpatched exploit in human cognition and uses that to its advantage. Which is not fun to be living in unless maybe if you are one of the people who owns the AI.
> And it also sounds like the opening chapter of a story about what happens when an AI finds a nice little unpatched exploit in human cognition and uses that to its advantage.
I vaguely remember an article arguing that companies are basically a form of AI, because in a way, there are processes and systems and the whole thing is more than the sum of the humans working in it or something.
Now replace "AI" with "company" in this sentence of yours and we are already there. The exploits being the gigantic slot machine of social media, notifications, short-form content and endless scrolling.
Forgive me if you’re already familiar, but if you’re interested in this metaphor you may like reading Stafford Beer’s work on organizational and system models. (1959, 1972)
Yeah the entire history of capitalism is basically a slow form of this and I'm not looking forwards to what happens when a bunch of giant bags of money pretending to be humans make all these bad decisions and brain-exploits happen even faster because it might make a little more money for them and who gives a fuck about the consequences?
Let's not forget people in very powerful places want to use AI to create a new "correct" religion. So this sort of thing does not surprise me in the least. I'll give it 2-3 years before we start seeing the public side of this work being done on the back end.
Most non-technical people have an inaccurate mental model of how LLMs work, based mostly on sci-fi movies like 2001 or Terminator, where they anthromorphise the model and assume that since it gives more accurate responses to most questions than any human they know that it must be some kind of superhuman genius master computer, sitting in the OpenAI headquarters pondering deep questions about reality.
Of course, when you go and finetune a model yourself, or run a smaller model locally, it becomes obvious that it doesn't have any mystical abilities. But almost nobody does this. I'm no anti-AI crusader, I use them all the time and think they're great and there's plenty of potential to use them to grow the economy, but the hype is insane.
It doesn't help when influencers like Yudkowsky go on anthromorphising about what it "knows", or go on with that mystical nonsense about shoggoths, or treat dimensionality of embeddings as if the model is reaching into some eldritch realm to bring back hidden knowledge, or hype up the next model talking about human extinction or anything like that.
It also doesn't help when the companies making these models:
- Use chat interfaces where the model refers to itself as "I" and acts as if it's a person
- Tune them to act like your best buddy and agree with you constantly
- Prompt the LLM with instructions about being an advanced AI system, pushing it toward a HAL or Skynet-type persona
- Talk about just how dangerous this latest model is
You tell these things that they're a superhuman AI then of course they're going to adopt all the sci-fi tropes of a superhuman AI and roleplay that. You tell them they're Dan the bricklayer they're going to act as if they're Dan the bricklayer.
> Most non-technical people have an inaccurate mental model of how LLMs work, based mostly on sci-fi movies like 2001 or Terminator, where they anthromorphise the model
That's a twisted perspective. The fact non-technical people misunderstand this technology can only be blamed on AI companies who are the first to anthropomorphize their products. Phrases like "thinking", "reasoning", "chain-of-thought", the term "AI" itself for that matter, let alone "AGI", are precisely what leads people to believe that these tools are far more capable than they actually are. We can't blame the general public for being misled by companies.
> These situations are obviously alarming. But positioning artificial intelligence as the primary culprit in these stories—as Eliezer Yudkowsky did in a tweet storm—is well, kind of lazy?
It's lazy, but IMO not for any of the subsequent narrative about media studies. It's lazy because the people were obviously already suffering from psychosis. ChatGPT didn't make them insane. They were already insane. Sane people do not believe ChatGPT when it tells them to jump off a roof! And insane people do insane things literally all the time. That's what it means to be insane.
The idea that ChatGPT convinced someone into being insane is, honestly, insane. Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot.
Psychosis is not necessarily something that you are or aren't; you can be prone to it, without it having manifested, and there can be external things that trigger it.
It isn't hard to imagine that a chatbot that seems to know a lot, and which can easily be convinced to produce text about any arbitrary subject with no grounding in reality, and which is prone to just making up plausible sounding text which is written in an authoritative tone, could be something that could easily trigger such psychotic episodes.
And it doesn't seem improbable that the interactivity of it, the fact that it responds to what is going on in someone's mind, could make it even more prone to triggering certain types of psychosis more easily than traditional unidirectional media like writing, TV, or radio.
Now, that's all supposition. For now, we just have a few anecdotes, not a rigorous study. But I definitely think it is worth looking into whether chatbots are more likely to trigger psychotic episodes, and if there are any safety measures that could be put in place to avoid that.
The non-o-series models from OpenAI and non-Opus (although I have not tried the latest, so it's possible that it too joins them) from Anthropic are cloyingly sycophantic, with every other sentence of yours containing a brilliant and fascinating insight.
It's possible that someone already on the verge of a break or otherwise in a fragile state of mind asking for help with their theories could end up with an LLM telling them how incredibly groundbreaking their insights are, perhaps pushing them quicker, deeper more unmoored in the direction they were already headed.
Psychosis is not a binary, and there are strong genetic components, but it's not the entire story. People develop psychoses where there weren't any before all the time, often but not necessarily in response to trauma. Some people are more predisposed to it than others.
Even if you hold that you need that genetic predisposition, and that a perfectly sane person can never become "insane" as a result of an experience (not a position I agree with), there is still the very real possibility that many people with the predisposition would never have had this "already insane" condition ever triggered or exposed. Think about the suicide cults that have formed, notably Jonestown. It's easy to consider the mass suicide (and murder of the members' own children) as "insane" behavior. What are the odds that every single person was genetically insane? Comparatively, what are the odds that an extremely persuasive and charismatic personality can rewire a person's reasoning to believe something "insane"?
If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.
> If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.
Thinking back on the documentaries about NXIVM, personas like Keith Raniere also seem eerily similar to LLMs in the sense that you can give them any prompt on the spot, and they will start generating long speeches which are essentially bullshit but on the surface seem coherent or even deep.
Another danger, that I guess we just have to live with now, is not just that ChatGPT could Jim Jones people, but that someone who didn't have the natural charisma of a cult leader could use ChatGPT on purpose to Jim Jones people. It may not work so well in person, but online ... yikes.
ChatGPT writes extremely persuasively for whatever claim you ask it to make - that's pretty much what it's designed to do! If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"?
> If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"?
Your example might sound ridiculuous, but this is actually happening all the time. People might not literally jump off a roof, but they do blow themselves up with bombs, go on killing sprees or commit suicide because someone told them so.
More importantly "sane" people are fully capable of unconciously making "insane" people jump of the roof. Its not even possible to count how many times that happens.
One proxy signal is counting the number of people who start crying about their "good intentions" or searching for thing to blame post facto.
Jurassic Park wasnt about Dinosaurs, it was about the limitations of the 3 inch chimp brain in controlling things that cannot be controlled.
The writer got overly stuck on one tweet and droned on and on about 15 words of a 45 word tweet. It was really poor research when there are other less reactionary statements she could have used.
Yudkowski’s point was that we’ve been operating under the assumption that generative AI is aligned by default with humans. Cases like this call that into question.
It's friction. It's easier to become addicted to gambling if you can gamble from your phone vs having to go to a casino or a bookie. It's easier to become addicted to drugs or alcohol if you're awash in drugs or alcohol. Prompting an LLM can mirror the same rush as gambling or gacha games. With ChatGPT, you have a "friend" who is never going to tell you to knock it off. "Psychosis" isn't an on/off switch.
Even among the more sane AI enthusiasts, I find that many have a weirdly millenialist zeal in their hot takes: getting defensive about its practical capability or reasoning capacity, hyperventilating about an imminent utopia/dystopia, etc.
Yeah, nothing ever happens. Except in the current day and age, decades happen in weeks, so the same happening with AI is not a stretch. I mean, just consider the capabilities of LLMs - if you look at it from a pre-COVID perspective, even knowing all limitations and strange failure modes up front, it's insanely impressive. To say that AI can contribute to a quickly arriving dystopia (not sure about utopia) is far from unreasonable.
Everyone knows we live in the Information age, not everyone realizes we're also living in an age Psychology - and of the two, we have a much clearer understanding of information and technology.
You can argue we're seeing a continuation of a pattern of our relationship with media and it's evolution, but in doing so you affirm that the psyche is vulnerable under certain circumstances - for some more than others.
I think it's a mistake to err on the side of casual dismissal, that anyone who winds up insane must have always been insane. There are well known examples of unhealthy states being induced into otherwise healthy minds. Soldiers who experience a war-zone might develop PTSD. Similar effects have been reported for social media moderators after repeated exposure to abusive online content. (Trauma is one example, I think delusion is another less obvious one w.r.t things like cults, scientology, etc.)
Yes, there are definite mental disorders like schizophrenia and bi-polar, there's evidence these conditions have existed throughout history. And yes, some of us are more psychologically vulnerable while others are more robust. But in the objective sense, all minds have a limit and are vulnerable under the correct circumstances. The question is a matter of "threshold."
I'm reminded of the deluge of fake news which, only a few years ago, caused chaos for democracies everywhere. Everything from Q anon to alien space ships, people fell for it. A LOT of people fell for it. The question then is the same question now, how do you deal with sophisticated bullshit? With AI it's especially difficult because its convincing and tailor made just for you.
I'm not sure what you would call this metric for fake news and AI, but afaict it only goes up, and it's only getting faster. How much longer until it's too much to handle?
Someone pointed this out in the comments section of the original post, but this was entirely missing the point Eliezer was trying to make, which was that these incidents serve as a narrow refutation of alignment-by-default claims, because either these LLMs aren't coherent enough to be meaningfully "aligned" at all, or they are, and are nevertheless feeding the delusions of many people who talk to them in ways we'd consider grossly immoral if done by a human. (One can quibble with whether these two possibilities cover all the available options, but that would be the argument to engage with, not with whatever the author understood EY to be saying.)
It does seem quite bad to me that either OpenAI can't, or won't, prevent their systems from following people into mania and psychosis. Whether it actually is causing the breaks from reality, idk. When you have CEOs selling these systems as robot friends, probably it's bad that said robot friends cheerfully descend into folies à deux fairly reliably.
LLMs are dangerous in that the basically mirror and mimic whatever you put in. Paranoia in, magnified paranoia out. A mostly stable person undergoing a temporary crisis could spiral down further after the model picks up on and reflects it back in an attempt to be helpful per its overarching command to be so.
And if a person is already unbalanced it could definitely push them off the cliff into very unhealthy territory. I wouldn't be surprised if the reported incidents of people thinking they're being gang stalked doesn't increase as model usage increases.
Let alone spiritual guidance and all its trappings with mysticism.
It can be helpful in some ways but you have to understand the majority of it is bullshit and any insight you gleam from it, you put there, you just may not realize it. They're basically rubber duckies with a keyboard.
Readit News
It's kind of hilarious if you ignore how much this is fucking up the relationships of the people this is happening to, similar to the way YouTube or Facebook loves to push people down a rabbit hole to flat-earth/q-spiracy bullshit because that shit generates Controversy. And it also sounds like the opening chapter of a story about what happens when an AI finds a nice little unpatched exploit in human cognition and uses that to its advantage. Which is not fun to be living in unless maybe if you are one of the people who owns the AI.
I vaguely remember an article arguing that companies are basically a form of AI, because in a way, there are processes and systems and the whole thing is more than the sum of the humans working in it or something.
Now replace "AI" with "company" in this sentence of yours and we are already there. The exploits being the gigantic slot machine of social media, notifications, short-form content and endless scrolling.
https://youtu.be/IwTq7LbUO6U
Of course, when you go and finetune a model yourself, or run a smaller model locally, it becomes obvious that it doesn't have any mystical abilities. But almost nobody does this. I'm no anti-AI crusader, I use them all the time and think they're great and there's plenty of potential to use them to grow the economy, but the hype is insane.
It doesn't help when influencers like Yudkowsky go on anthromorphising about what it "knows", or go on with that mystical nonsense about shoggoths, or treat dimensionality of embeddings as if the model is reaching into some eldritch realm to bring back hidden knowledge, or hype up the next model talking about human extinction or anything like that.
It also doesn't help when the companies making these models:
- Use chat interfaces where the model refers to itself as "I" and acts as if it's a person
- Tune them to act like your best buddy and agree with you constantly
- Prompt the LLM with instructions about being an advanced AI system, pushing it toward a HAL or Skynet-type persona
- Talk about just how dangerous this latest model is
You tell these things that they're a superhuman AI then of course they're going to adopt all the sci-fi tropes of a superhuman AI and roleplay that. You tell them they're Dan the bricklayer they're going to act as if they're Dan the bricklayer.
That's a twisted perspective. The fact non-technical people misunderstand this technology can only be blamed on AI companies who are the first to anthropomorphize their products. Phrases like "thinking", "reasoning", "chain-of-thought", the term "AI" itself for that matter, let alone "AGI", are precisely what leads people to believe that these tools are far more capable than they actually are. We can't blame the general public for being misled by companies.
You could still get the same information out of them, but you had to be creative to come up with a relevant "document" stub that it would complete.
It's lazy, but IMO not for any of the subsequent narrative about media studies. It's lazy because the people were obviously already suffering from psychosis. ChatGPT didn't make them insane. They were already insane. Sane people do not believe ChatGPT when it tells them to jump off a roof! And insane people do insane things literally all the time. That's what it means to be insane.
The idea that ChatGPT convinced someone into being insane is, honestly, insane. Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot.
It isn't hard to imagine that a chatbot that seems to know a lot, and which can easily be convinced to produce text about any arbitrary subject with no grounding in reality, and which is prone to just making up plausible sounding text which is written in an authoritative tone, could be something that could easily trigger such psychotic episodes.
And it doesn't seem improbable that the interactivity of it, the fact that it responds to what is going on in someone's mind, could make it even more prone to triggering certain types of psychosis more easily than traditional unidirectional media like writing, TV, or radio.
Now, that's all supposition. For now, we just have a few anecdotes, not a rigorous study. But I definitely think it is worth looking into whether chatbots are more likely to trigger psychotic episodes, and if there are any safety measures that could be put in place to avoid that.
It's possible that someone already on the verge of a break or otherwise in a fragile state of mind asking for help with their theories could end up with an LLM telling them how incredibly groundbreaking their insights are, perhaps pushing them quicker, deeper more unmoored in the direction they were already headed.
Even if you hold that you need that genetic predisposition, and that a perfectly sane person can never become "insane" as a result of an experience (not a position I agree with), there is still the very real possibility that many people with the predisposition would never have had this "already insane" condition ever triggered or exposed. Think about the suicide cults that have formed, notably Jonestown. It's easy to consider the mass suicide (and murder of the members' own children) as "insane" behavior. What are the odds that every single person was genetically insane? Comparatively, what are the odds that an extremely persuasive and charismatic personality can rewire a person's reasoning to believe something "insane"?
If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.
Thinking back on the documentaries about NXIVM, personas like Keith Raniere also seem eerily similar to LLMs in the sense that you can give them any prompt on the spot, and they will start generating long speeches which are essentially bullshit but on the surface seem coherent or even deep.
Your example might sound ridiculuous, but this is actually happening all the time. People might not literally jump off a roof, but they do blow themselves up with bombs, go on killing sprees or commit suicide because someone told them so.
If a person had convinced somebody to kill themselves then they would be, rightly, prosecuted for it.
Dead Comment
One proxy signal is counting the number of people who start crying about their "good intentions" or searching for thing to blame post facto.
Jurassic Park wasnt about Dinosaurs, it was about the limitations of the 3 inch chimp brain in controlling things that cannot be controlled.
Yudkowski’s point was that we’ve been operating under the assumption that generative AI is aligned by default with humans. Cases like this call that into question.
Deleted Comment
The Joe Rogan of AI.
I am on team boring. AI is technology.
You can argue we're seeing a continuation of a pattern of our relationship with media and it's evolution, but in doing so you affirm that the psyche is vulnerable under certain circumstances - for some more than others.
I think it's a mistake to err on the side of casual dismissal, that anyone who winds up insane must have always been insane. There are well known examples of unhealthy states being induced into otherwise healthy minds. Soldiers who experience a war-zone might develop PTSD. Similar effects have been reported for social media moderators after repeated exposure to abusive online content. (Trauma is one example, I think delusion is another less obvious one w.r.t things like cults, scientology, etc.)
Yes, there are definite mental disorders like schizophrenia and bi-polar, there's evidence these conditions have existed throughout history. And yes, some of us are more psychologically vulnerable while others are more robust. But in the objective sense, all minds have a limit and are vulnerable under the correct circumstances. The question is a matter of "threshold."
I'm reminded of the deluge of fake news which, only a few years ago, caused chaos for democracies everywhere. Everything from Q anon to alien space ships, people fell for it. A LOT of people fell for it. The question then is the same question now, how do you deal with sophisticated bullshit? With AI it's especially difficult because its convincing and tailor made just for you.
I'm not sure what you would call this metric for fake news and AI, but afaict it only goes up, and it's only getting faster. How much longer until it's too much to handle?
And if a person is already unbalanced it could definitely push them off the cliff into very unhealthy territory. I wouldn't be surprised if the reported incidents of people thinking they're being gang stalked doesn't increase as model usage increases.
Let alone spiritual guidance and all its trappings with mysticism.
It can be helpful in some ways but you have to understand the majority of it is bullshit and any insight you gleam from it, you put there, you just may not realize it. They're basically rubber duckies with a keyboard.