We need to push the notion that "closed-source LLMs are super dangerous, an existential risk to humanity".
Basically we need to equate "safety" in LLMs to mean "being open-source".
OpenAI keeps talking about "safety" as the most important goal. If we define it to mean "open-source" then they will be pushed into a corner.
We are at a critical time period that will literally decide the outcome of humanity.
The fear of large corporations controlling AI is an argument against regulation of AI. Regulation will guarantee that only the biggest, meanest companies control the direction of AI, and all the benefits of increased resource extraction will flow upward exclusively to them. Whereas if we forego regulation (at least at this stage), then decentralized and community-federated versions of AI have as much of a chance to thrive as do the corporate variants, at least insofar as they can afford some base level of hardware for training (and some benevolent corporations may even open source model weights as a competitive advantage against their malevolent competitors).
It seems there are two sources of risk for AI: (1) increased power in the hands of the people controlling it, and (2) increased power in the AI itself. If you believe that (1) is the most existential risk, then you should be against regulation, because the best way to mitigate it is to allow the technology to spread and prosper amongst a more diffuse group of economic actors. If you believe that (2) is the most existential risk, then you basically have no choice but to advocate for an authoritarian world government that can stamp out any research before it begins.
There's nothing stopping regulation of closed-source commercial AI, while promoting academic and opensource use conforming to ethical guidelines.
These two things are orthogonal, and need to be considered seriously and separately: how to ensure that LLMs are not enabling serious abuses, through various kinds of regulation, and also how to ensure that our common usage of LLMs is not gatekept by huge corporations and dependent on their goodwill and continued interest in supporting the particular things we want to do with them.
All this brings me back to the Vulnerable World Hypothesis proposed by Nick Bostrom. In fact global totalitarian surveillance that stamps out all potentially dangerous research is one of his few proposed solutions in his paper. I don't know if I can stomach such a solution, but I think we are living the early stages of such a world that forces such questions to be asked, not one in which we picked out a "black ball" that will make us go extinct, but a sort of "green ball" that irreversibly poisons specific parts of collective humanity.
I would personally be in favor of OpenAI keeping GPT-4 and future models proprietary, much as it won't affect the open source spirit. Random hackers with powerful GPUs generating terabytes of SEO spam are less visible to the world and more difficult to hold publicly accountable than a huge corporation with billions in funding.
Have a read of this write up, it compliments this discussion nicely.
I think it makes some good points, but one of them stands out clearly to me, there is a narrative going around that we "have" to continue doing AI research at this pace. I'm skeptical about it. This is a story we believe.
We're fueling an arms race by doing the research, do you know what I mean? Like if we were throwing billions of dollars into gray goo creating technologies, would that be clever?
I'm not saying that it's possible to slow down or stop AI progress, but we're definitely not helping by throwing billions or trillions of dollars into it.
If we were an actually intelligent species, I think we would slow down and take stock of what we're doing before disaster strikes.
In fact global totalitarian surveillance that stamps out all potentially dangerous research is one of his few proposed solutions in his paper.
Someone else posted this, I actually think this is where further AI research will take us, this is because regular people who aren't nerds will vote for this, it would be politically an easy sell for a politician, and people would honestly prefer oversight and their families safety rather than having watching "boffins gone wild" receive endless rounds of funding creating self-replicating terminators for fun.
You have to remember, computer geeks and regular people probably have very different ideas about AI. If we can't behave, people will behave us.
Deleted Comment
But one other reason I think this is flawed is that it assumes innovation in those models has finished. We're not at the stage where these are good enough to revolutionise everything. There's a lot more research, hard work and creativity that needs to be unleashed for all benefits to be realised. Traditionally, for-profit startups have been the best vehicle for that to happen. OpenAI has only scratched the surface and can do a lot more. Forcing them into open-sourcing and only caring about safety would quickly stop this progress. They are not a megacorp extracting rent who we need to fight - they are literally a startup changing the world in front of our eyes.
But this isn’t about mega-corp vs startup who is better at changing the world.
There is a potential major civilization-ending downside to the amount of change true AGI could bring before we could control it. Or at the very least an unaccountable central autocrat who will own the entire world if they can control the AGI.
OP is suggesting open source is one way to shine sunlight on the innovations that are happening that we will all be affected by and thus should have a voice in controlling.
Deleted Comment
Just think about what a mega-corporation like Microsoft, whose primary interests lie in accumulating capital and market control, can do with this technology that is essentially embedded into everything. Of course, there are potential benefits, but I'm personally more interested in the risks.
With LLMs, I understand why people look very negatively on the prospect of them being controlled by major corporations - certainly there are huge issues with that. But if we believe that they're going to develop world-ending capabilities, I would rather they be in the hands of wealthy, powerful corporations that are strongly incentivized not to destroy the world, vs. being freely available to people who want to burn it all down.
Mutually assured destruction works because the only people who are party to it are those who don't want to be destroyed. If everyone has access to world-ending tech, it only takes one person who doesn't care to end it all. In that sense, asymmetry is very good.
In comparison it's trivial for both terrorists and powerful nation-states to download 60GB of weights and crunch numbers on a GPU cluster. Arguably the hardware requirements of the most sophisticated LLMs act as a similar barrier to widespread adoption even if they were open-source, but to nowhere near the same extent, and as with nukes that's just a happy coincidence of the current state of technology/known physical limitations than an intentional barrier to stopping their spread. Even as we speak there has been collective interest in discovering a way to decrease those hardware requirements (INT8/INT4 quantization) as much as possible so more people can run them for themselves, which shows no signs of abating.
All I can say is we're lucky that physics prevents us from mass-producing ICBMs on a fast enough time schedule, because if it were possible we'd probably attempt to do so for the sake of trying.
The magic in the software isn't really the software as much as the training data. How does being able to see the source code give you any insight in terms of what data it is trained with, or what people are doing with it?
We can't trust that openAI is not creating risks
A few ideas:
Impersonation at scale. i.e. everyones image and text and sometimes video is freely available one sites like HN, Reddit and social media. Anything that requires voice authentication (calling friends, etc) is now not a thing anymore if you have videos of yourself talking for more than a few minutes online (IG, Facebook, interviews, someone recording you at work, etc)
Targeted impersonation. i.e. Train a model on politicians, local ones are better, and disrupt whatever you want by sending a video to news outlets (see the twitter hack for a bit ago where peoples accounts got taken over. Can do that with your face and voice now. Zoom, press releases, etc.)
Social engineering at scale. i.e. any text you see could be written by an model (such as your comment) and so any information you give out could be to a bad actor. Situations like DM's, etc. Those nigerian scams now just got a lot more effective.
ddos and things like that can be fluid and perpetual. "Run this attack, if it stops working, change a few parameters until works again"
Deepfake blackmail. i.e. change a pornstars face to be one of a girl/guy at your highschool and tell everyone you are going to send it out if you dont do XYZ. Even though it's not you, it looks identical so no one will believe you when you say it's not.
These are just a few off the top of my head and I am not that creative either.
I can't even imagine a worst thing at this point, but plenty of bad things, like a mass robocalling campaign to convince your grandma to send your money using your voice and likeness if needed.
Scale of bad actors actions is a quality in itself.
Look at how Google's stock price got wrecked when a demo went bad. Or how Bing had to make emergency changes to their chatbot after somebody provoked it into behaving belligerently. (Funny I demoed my A.I. to a group for the first time this morning and had no fear it would go wrong because I've used it every day since Dec 27)
If you are the first or second or third chatbot and it gets out that some sicko got it to write something sick then it is a big deal. If this is the 572th chatbot then it isn't news anymore.
As for "dangerous devices" that is surely overrated too. It's hard to make an effective bomb and you're the first person you're likely to blow up. Ted Kaczynski took nearly 20 years to learn how to make reliably deadly bombs. Dzhokhar Tsarnaev attended one of the best terrorist training camps in the world. The Morewrong prophet is afraid an LLM will spit out a DNA sequence that some rando will send to a mail order DNA synthesis firm to make a 100% deadly virus but it would take quite a bit more than that (maybe thousands of DNA sequences, a very good lab, and some highly trained researchers who aren't afraid to be killed by what they are working on) and if you had those resources you could do "gain-of-function" experiments and make very dangerous viruses without the LLM.
Long term what bothers me is not what goes on in public but what goes in private, maybe
https://abcnews.go.com/Technology/intimate-ai-chatbot-connec...
for instance we know people will follow a prophet like Jim Jones and commit suicide or that a follower of an blackpill incel could become a mass shooter, but all of those things go on in public. The CIA learned in MKULTRA that there is no magic pill or technique that can take the average person and make them into a Manchurian Candidate but if you're willing to wait for the right victim you can very much lead an isolated and directionless person down a rabbit hole.
For instance after 9/11 the FBI tried to bait a Muslim into a fake terrorist attack with a "Sting" operation and couldn't do it. Instead they found some poor black guy who thought Malcolm X was cool and he was so impressionable they were able to give him a fake bomb and fake guns and rent a synagogue for him to "blow up" and he was shouting "Allah Akbar!" when the police came for him and had no idea he was set up and knocked down like a bowling pin.
The system of a chatbot + a vulnerable human could go into very dangerous places whether or not the chatbot is specifically programmed to do that or through the mutual process of "reinforcement learning".
Now maybe that's like the fear that listening to rock music is going to make you become a drug addict or become pregnant, but it's a concern that will come up. See
https://abcnews.go.com/Technology/intimate-ai-chatbot-connec...
of course this is not something fundamentally new, as some people think the parasocial relationships that people develop on OnlyFans are dangerous and plenty of "Freezone" Scientologists have telepathic conversations with L. Ron Hubbard although it seems to me it really would be a gas to teach a chatbot to write like Ron and specifically complete all the O.T. levels that he never got around to finishing.
I don't know about you, but I definitely don't trust Facebook, Google, Microsoft, and OpenAI to be either altruistic or transparent with their usage of these LLMs.
The genie's already out of the bottle. The best move now is to work on rapidly improving the quality of the datasets that these models are based on to ensure the best possible outcomes.
https://www.goodreads.com/book/show/2735486-eco-computer
debunks the scenario of this movie
https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project
(I try not to post Youtube clips but you really have to see the trailer for this one if it is available in your geography)
https://www.youtube.com/watch?v=kyOEwiQhzMI
and makes the case that there is not going to be one unitary "breakout" of one genius machine but rather we are gradually going to lose control of a network of competing and uncoordinated intelligences and suffer natural disasters somewhat like hurricanes as opposed to being killed off or enslaved. The "Love Letter" crisis was one of the first of these.
We "trust" NotOpenAI?
I strongly suspect that LLMs will not be the miracle (or disaster) that they're currently being sold as.
There was a time I was doing all sorts of sales and networking calls and found there were a lot of people in the Research Triangle Park area who had forgotten more about text analysis than anyone in the Bay Area knew.
The "secret" was not one great idea such as transformer models but rather the social and technical organization to break up difficult problems into easy and difficult parts and farm those out to very large groups of people. A person getting started in text analysis might look at Apache UIMA and ask "Where's the beef?" because UIMA doesn't really do anything all but rather is a framework for getting 10,000 people pulling in the same direction. IBM Watson did have the "big idea" of using probability scores to combine multiple question answering strategies but the real "secret" again was that they had the equivalent of an army batallion work on it.
If you don't believe me, try driving from the BWI airport to Washington DC at rush hour and witness for yourself America's worst traffic jam when the day shift at Ford Mead (the NSA headquarters) gets off work. If there's one "secret" about secrets it that it gets increasingly hard to keep a secret the more people that know about it.
(And I'll say yes, I know those details make me sound paranoid, I am just as paranoid as I sound, because if I told you I wasn't paranoid that would just convince you all the more that I am paranoid.)
I am 99.9% sure that this is the case. The hype level is insane right now, and people (both pro and anti) are commonly asserting patently absurd things as fact.
In the end, I predict, we'll find that this is a powerful and useful tool, but it won't be earth-shaking.
How?
> I strongly suspect that LLMs will not be the miracle (or disaster) that they're currently being sold as.
Why do you think this? See Heuristics That Almost Always Work: https://astralcodexten.substack.com/p/heuristics-that-almost...
Oh no, anything but a corner!
Why would they care?
Yes, it's that depressing.
How do we know it's the "current public discourse"