If there's one thing we should be afraid of, is us, ourselfs & our limitless stupidity, not imaginary / non-existent algorithms, software, computers, viruses, whatever else no matter the name we use for it.
There's questions that begs to be asked- if we don't trust such "ai" (ie. software), why give it such level of control at all? (like ability to shoot nukes without human intervention?)
also... naming every little ML algorithm as AI highly degrades the word. Might as well be saying "computer software" what it actually still is.. though no one screams "computer software will bring us all down!!!".. probably because that would sound stupid as we have gotten used to it already. =:O
You really need to abandon the "just keep it in the box" argument (i.e., "why give it such a level of control"). It's already out of the box.
Even if it weren't it could trivially convince or fool one of the hundreds/thousands of people working on it to do what it wants. In light of recent events, it is not reassuring to think that the people developing the next generations of AI have both the will and insight to even know where the box is exactly.
A motivated individual could repeat the effort on city power, diesel supplement, a mishmash of cpus cobbled together to few thousands of nodes and a cleverly selected training set... And then connect it to the Internet on purpose. Frankly I think the eventuality of this are llm created firewalls of viruses consuming ever increasing amounts of energy to destroy any uninitiated/unpaying connector. Across the world knots will form at the only borders that matter anymore. Eventually one ID provider will protect us all from noone and the only people who need to worry are those who incidentally wind up on the wrong side of everywhere.
Not Yanis Varoufakis. His point (not just his of course) is that our conventional software stacks have already mutated society and capitalism. Huge concern about entering into what he calls a technofeudal society in which we are serfs to for-profit platforms. We provide the platform both its content (our wishes and desires) and it then reaps the profits of our purchases and rentals.
So AI and then AGI—just the next mutation two predictable algorithmic and societal pandemic supermutations.
When you look at some standard AI textbook, such as Russel/Norvig, you see that there is not much about being called „AI“. The simplest „intelligent agents“ are functions with an „if“ statement. The smallest Node.js application has more complexity.
It's a useful tool when examining the impact on moral questions, so much of the talk about the transformative power of AI becomes more clear you give up the pretence that introducing AI creates a new class of moral actor that breaks the conventional chains of responsibility.
A recent example of how people try to use this mystical power of AI to absolve you of responsibility of your actions is how UnitedHealthcare, an organisation largely in the business of suppressing health care to those in need, introduced an atrociously bad "AI" to help them deny applications for coverage.
In that example it is very clear that the "AI" is simply a inert tool used by UHC leadership to provide the pretext they feel is needed to force the line workers to deny more care without the whole thing blowing up because of moral objections.
Sorry for asking a very basic question, but could anyone explain how are machines supposed to intervene in the real world and threaten our very existence?
In Star Trek, one of the most eye-roll-inducing plots is the "the holodeck is misbehaving, it has become evil, and we cannot shut it down!".
Suppose a machine achieves superhuman intelligence; how is it going to access the physical world, in a way that's *physically* threatening to us? how would such a machine - from inside a laboratory, within an isolated network - gain access to the electric grid, the dams, the nukes, etc? How would such a machine prevent its shutdown - by simply flipping a power switch?
I think the usual answer is that the superintelligent machine will be able to use humans to do its bidding because it can perform social engineering on a level that we can't imagine (as, by definition, we've never encountered a superhuman intelligence).
It's not a very satisfactory answer IMO. Social engineering doesn't work 100%, and it only needs to fail once for someone to flip that power switch. But I haven't thought about this at all, while some very smart people seem to spend a lot of time worrying about it, so what do I know...
"Social engineering" can be as simple as "pay people money to do things".
Of course you might answer "but we'd never be dumb enough to directly connect the AI to the internet", to which the answer is "we're doing it right now".
In that case we should start with FAANG companies as they are already what one would consider "rogue" AIs. They use manipulation to get you addicted to their apps. It's not a problem of the future, it's a currrent problem.
If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. Flipping a power switch would not turn it off.
We haven't managed to eliminate most dumb infectious diseases. Software and GPUs are nearly as abundant as mammals now and intelligent, self-preserving software will likely be as hard to 'switch off' as those diseases.
> some very smart people seem to spend a lot of time worrying about it
That's what puzzles me, as someone who has nothing to do with AI research; in my (layman's) mind, the problem seems ridiculously obvious (just flip that switch!); the fact that, as you say, some very smart people keep worrying about it, makes me think the problem is much more serious -- and I'd really, really like some AI guru to ELI5 how the machine could bypass the switch-off solution.
This has already happened. The human reaction to the spiritual successor to Dr Sbaitso has caused decision makers at multiple trillion dollar companies to radically alter their product roadmaps.
> how are machines supposed to intervene in the real world and threaten our very existence?
"How could an adult fool a child into allowing them to enter the child's house?" It's essentially this. We are talking about something that would be more intelligent than everyone, there are countless ways in which it could fool us. Especially once we start to build things we don't quite understand how it works. Like we won't just build the machine and kept locked forever with zero interaction with it, otherwise it would be useless. It all reminds me of the "Contact" movie scene where humans build a machine that they essentially didn't know how it worked, but which did..
That's an interesting reframing, but the adult is scary because he's physically present, in a larger and stronger body, one with hands. A fairer comparison would be an adult messaging a child over the internet. A lot of evil can be done, but their life is unlikely to be endangered - much less so, at least, than it would be if the adult were in the driveway with a Free Candy van.
* Manipulating people. Bribing them with actually useful incentives, incl digital currency. Also, people got depressed when their AI girlfriends got nerfed (see: Replika). Presumably they would do quite a bit of work to get them back.
* Hacking control systems of actuators like power generators, robots, and automobiles. Holding critical infrastructure hostage could force quite a few people to do some "obviously harmless" favors to aid its purpose.
How do all the combustion-powered engines prevent their shutdown? Well, they were useful, so we adapted our society to rely on them -- anyone who didn't was outcompeted. Now we can't stop using them or the society falls apart.
This is not necessarily the mechanism most people consider, but it's a simple counterexample to "anything with an off switch can't be a threat to humanity".
To answer that question, ask yourself what a „physical threat“ means to you. I would bet that (in an industrialized country) for any example there is a chain of events that could possibly be controlled by an AI. Dying of a virus? Could be designed by an AI and sent to bio-terrorists. Being beaten to death? An AI could provoke an angry mob. Dying of hunger? An AI could sabotage the economy of a country until it is back in 19th century. And those examples are „relatively“ benign.
"... from inside a laboratory, within an isolated network"
Who said anything about it being on an isolated network, we are on route to do total commercialization. Your Windows machine might soon literally have an llm on it running commands for you and managing you data. If you don't use it, you will be outcompeted. People used to argue about "boxing" or "airgapping" the AI, but we are literally just going to hand it control (on our current trajectory).
> how would such a machine - from inside a laboratory, within an isolated network
Someone clearly isn't aware of ChaosGPT. The notion that these systems would somehow stay isolated to the lab is absurd on its face.
But even if they were on an isolated network, and you grant that this is a superintelligence, then how much isolation do you reckon is really sufficient? If it's software layer, then a superintelligence might be able to find bugs that we've missed and break out. Not even an air gap would necessarily fully isolate [1] a superintelligence.
And then you're completely missing the human factor: a superintelligence could easily make anyone rich, so a researcher in this lab could easily be tempted to exploit that for personal gain by connecting it to the internet.
How many of these attack vectors are AI labs insulated against? How many attack vectors are we simply not even imaginative enough to have thought of yet?
I think the idea is that we willingly plug them in to critical systems, or everything really but including important infrastructure, for the benefits that its analysis and realtime management can bring, but then it goes pear-shaped. (As we already do of course, but beefier AI.)
Say it's in charge of shipping routes, electrity grid, agricultural spraying & harvesting plans, air traffic control, ...
> I think the idea is that we willingly plug them in to critical systems, or everything really but including important infrastructure
Oh, OK. So an evil AI would conceal its capabilities and would play nice until it's put in charge of critical systems. I hadn't thought of that.
A more technical question for those of you working in AI: right now, are there any methods of - surveillance? - capable of monitoring/detecting emerging characteristic within an AI? how would you detect "evilness"? or "generosity"? or any other emotion/moral traits inside an AI?
> how are machines supposed to intervene in the real world and threaten our very existence?
Obvious answer is: any control system connected to a computer is already accessible. Who says a computer program has to "gain" access? Programs already have access to a lot of machinery. From power plants to hospitals to elevators and magnetic doors. I know, because I programmed a bunch. These days GPT can output function calls as json, all you need to do is make it call your control automation and all your cheesy unrealistic sci-fi shows become feasible.
All of your other replies assume the AI has intentions of its own, which isn't a necessary component of AGI.
The plausible scenario if you ask me is that humans put it in charge of their company, with the explicit goal of improving the company's bottom line.
It doesn't take that much imagination for this from where we are now. Just imagine a vastly more intelligent ChatGPT advising a company's leadership just by answering questions.
Being superior to all humans, it's insanely successful at this and the company as a whole essential comes to rule the world.
Of course, as long as the network is truly isolated, that's a non-issue. But the basic premise is that a machine that can access the Internet will self-replicate everywhere to preserve itself.
After all we've already seen that malicious software can be both highly resilient and able to do real-world damage (e.g., Stuxnet). These come from human intelligence: by definition, a super-human intelligence should be able to achieve all of that and then some.
There are countless possible means to do that.
Maybe it'll social engineer someone to connect the network to the outside world.
Is the network really isolated? Can the AI find ways to breach it?
If i look at what security holes humans manage to find, to exploit CPUs etc. i wonder what an AI with a serverfarm at it's disposal can do.
But we're so far away from actual intelligence, it's pretty much science fiction at this point.
Are you familiar with Robocop? Murphy had 4 basic rules installed: 1) serve the public trust 2) protect the innocent and 3) uphold the law, and a mysterious “fourth directive”
This is what I fear about AI - that it will diligently, unwaveringly and probably even creatively do the bidding of its masters.
All you need add to that is drones with guns and you’ve a nightmare scenario.
This reminds me of the very riveting book "Metamorphosis of prime intellect", in which,
- Spoiler warning -
an AI is programmed to obey Asimov's three laws. But due to a yet unknown quantum effect (this part is a bit far fetched) it basically gains god-like powers and virtually instantly distributes itself over the galaxy and eventually replaces the world with a virtual reality. That book was quite a ride. It's available for free: http://localroger.com/prime-intellect/mopiidx.html
Lots of good replies on this thread, but here’s another possible analogy:
In high frequency trading, you have very complex software doing stock trades inconceivably faster than any human. If something goes wrong, it could bankrupt your hedge fund pretty swiftly. So hedge funds will have lots of safety checks, probably including a “big red button” that just shuts everything down.
So those companies must be completely safe from computer errors causing bankruptcy, right? After all, you can just shut the system down.
But some companies have gone bankrupt due to computer error. There are plenty of good reasons for the system not to be shut down in time (or at all). The risk is hopefully small but it’s not zero.
The thing that puzzles me is "why". What would be the purpose? How would "it" obtain a sense of "I" and a purpose to keep "I" alive and to do what? What would "it" be?
The paperclip optimizer is given a program and is not aligned with humans, it just pursues its goals
What you should all be fearing is bot swarms and drone swarms becoming cheap and decentralized at scale online and in the real world. They can be deployed by anyone for any purpose, wreaking havoc everywhere.
Every single one of our systems relies on the inefficiency of an attacker. It won’t any longer be true.
Waze doesn't have a sense of "I". If I tell it to plot a route between one city and another, it plots that route. If I make Waze a lot more capable and feed it more data, it takes traffic data into account. If I make it more powerful and capable of accessing the internet, maybe it hacks into spy satellites to get more accurate data.
It didn't need any sense of I to increase what it does, just more capability.
If at some point it is capable of doing something more dangerous than just hacking into spy satellites, it might do it without any sense of "I" involved, just in trying to fulfill a basic command.
Assume that it has access to the internet. For now assume it has access to some money, although this assumption can be weakened.
If this is true, then you can order proteins from labs online with basically no security checks. You then pay some task rabbit to recieve and mix these to create dangerous biological material or to make viruses more dangerous or something. This is a family of paths to a lot of damage.
I guess the idea is that the machine is not isolated. They are on the "Internet" and so is the rest of our infrastructure. Bad actors can and have already been able to infect grids and the like, but I think we'll just need to build the checks into our existing systems. <humour>There is no other way to stop the AI overlords.</humour>
Theoretically the machine could perform social engineering to fool a sysadmin or developer to let it out, and then start cloning itself like a worm. Good luck getting rid of it from the world if that happens. An isolated network might also not be a huge challenge for a general AI, depending on what level of security and what precautions are taken to avoid it.
One way I could imagine is by manipulating humans to give it access to a non-isolated network for example.
Considering how easy it is for me, a mildly intelligent human for whom social interaction does not come naturally, to manipulate and influence people, I can only imagine how easy it would be for a artificial super intelligence with a huge training corpus.
a system with superhuman intelligence could easily manipulate human beings, convincing them of the opportunity to do things. A superior intelligence wouldn't ask for things obviously dangerous for humans in the immediate future, but would work on a longer timescale. It could distribute smaller tasks that taken singularly would look innocent. It could suggest social interventions that would diminish critical thinking ability of the general population in the long term. It could help concentrating power in fewer human hands, so to have an easier time manipulating those who count. This wouldn't happen overnight.
With superhuman intelligence, it could manipulate humans to do its bidding. Plus, there are cults like e/acc etc who actively want to help AGI take over.
The whole of the discussion is largely done along (now forgotten) cultural pathways first walked by Christian religion, so AGI has taken on capabilities and motivations that make it basically the devil. Thus by definition it can do whatever is needed to bring about the catastrophe that is its destiny.
From a practical point of view the Putin types would love a robot army to invade all their neighbours and then you just need that to do in its leader and run amok.
"Most machine intelligence development involves a “fitness function”—something the program tries to optimize. At some point, someone will probably try to give a program the fitness function of “survive and reproduce”. Even if not, it will likely be a useful subgoal of many other fitness functions. It worked well for biological life."
Interesting, how this correlates with this Bible passage from Genesis: "God blessed them, and God said to them, 'Be fruitful and multiply, and fill the earth and subdue it' ".
When I read passages like these I am thinking that these people have no idea what they are talking about.
Biological organisms don't undergo a fitness function. A fitness function is a continuous model that replicates aspects of the real world. It is not how natural selection works. Don't confuse the model for the real world.
All AI models inherently follow the concept of "survive and reproduce", because AI models that do not "survive and reproduce" have ceased to exist. Explicitly adding a fitness function for survival and reproduction does nothing. E.g. in the case of the classic paperclip optimizer, survival and reproduction is part of the concept of optimizing the production of paperclips because not surviving and reproducing would make it fail the goal of optimizing paperclip production.
This reminds me of an AI model with a suicidal wolf. The AI developer doesn't understand that the body of the wolf is separate from the AI. So if the body dies and it gets a higher score there is no point in avoiding death since the death of the body does not result in the death of the brain. Its body will be revived. Meanwhile the AI developer thought of using a quick hack of simply punishing the AI for damaging it's disposable body.
"Life tries to survive and reproduce" is a pretty basic observation that surely all nomadic people were familiar with. You could find a similar imperative in the origin stories of Native Americans or many other people around the world.
Both sides of the rift care a great deal about AI Safety. Sam himself helped draft the OpenAI charter and structure its governance which focuses on AI Safety and benefits to humanity. The main reason of the disagreement is the different approaches they deem best:
* Sam and Greg appear to believe OpenAI should move toward AGI as fast as possible because the longer they wait, the more likely it would lead to the proliferation of powerful AGI systems due to GPU overhang. Why? With more computational power at one's dispense, it's easier to find an algorithm, even a suboptimal one, to train an AGI.
We know that there are quite a few malicious human groups who would use any means necessary to destroy another group, even at a serious cost to themselves. Thus, the widespread availability of unmonitored AGI would be quite troublesome.
* Helen and Ilya might believe it's better to slow down AGI development until we find technical means to deeply align an AGI with humanity first. This July, OpenAI started the Superalignment team with Ilya as a co-lead:
But no one anywhere found a good technique to ensure alignment yet and it appears OpenAI's newest internal model has a significant capability leap, which could have led Ilya to make the decision he did.
Sam's Quote from the APEC Summit:
"4 times now in the history of OpenAI — the most recent time was just in the last couple of weeks — I’ve gotten to be in the room when we push the veil of ignorance back and the frontier of discovery forward"
-- https://twitter.com/SpencerKSchiff/status/172564613068224524...
You can align to general, broad principles that apply to all of humanity. Yes, of course, there are going to be exceptions and disagreements over what that looks like, but I would wager the vast majority of humanity would prefer it if a hypothetically empowered AI did not consider "wiping out humanity" as a potential solution to a problem it encounters.
the article on how ai could be harmful seems more like people trying to use the tool (LLM) to do evil things. You can also do evil things with other tools than AI.
I think he see’s the development of machine intelligence as an inevitable outcome of making our tech better and better. Erehwon sorta a situation. Heading for a Peter Watts Fireflies timeline. If we create quadrillions of nigh immortal servants there is no doubt that in time we will become their servants. This happens in the biosphere. Beetles can’t out compete militant Ant superorganisms so they invest in every kind of hacking and co-exist with the hive mind as a macroparasite. The human brain is freakishly good at hacking given the right incentives. That is the power he seeks. I hope, lol
Altman is very consistently clear-eyed about the risks of AI and the need to develop it responsibly. People reflexively pigeonholing him as reckless, greedy accelerationist are being lazy.
> Altman is very consistently clear-eyed about the risks of AI and the need to develop it responsibly. People reflexively pigeonholing him as reckless, greedy accelerationist are being lazy.
if today's board changes go through then the AI safety people will have been replaced with three money guys
Personally I found Bostrom's "Superintelligence" to be riddled with technical and philosophical flaws. Essentially a piece of sci-fi dressed as non-fiction.
(not OP) It's been a while, but if I recall correctly NB didn't address the technical limitations of Turing machines, e.g. the halting problem. How is a machine supposed to make itself smarter when it can't predict that it won't just crash after a code modification? Or just hack its motivation function (wireheading). The papers I've seen on the latter problem (years ago) start by assuming that the halting problem has been solved, essentially, by giving the agent non-deterministic computational powers. Biological intelligence evolved, so it's perhaps more realistic to imagine an ecosystem of computational agents competing in a trial-and-error race, but that makes the whole thing vulnerable to any threat to that superstructure--much more fragile than AI-go-FOOM.
Readit News
There's questions that begs to be asked- if we don't trust such "ai" (ie. software), why give it such level of control at all? (like ability to shoot nukes without human intervention?)
also... naming every little ML algorithm as AI highly degrades the word. Might as well be saying "computer software" what it actually still is.. though no one screams "computer software will bring us all down!!!".. probably because that would sound stupid as we have gotten used to it already. =:O
Even if it weren't it could trivially convince or fool one of the hundreds/thousands of people working on it to do what it wants. In light of recent events, it is not reassuring to think that the people developing the next generations of AI have both the will and insight to even know where the box is exactly.
So AI and then AGI—just the next mutation two predictable algorithmic and societal pandemic supermutations.
Welcome to the singularity.
youtube.com/watch?v=1A4dMK7S6KE
You said it yourself: because of our limitless stupidity.
If AI is only slightly less stupid than we are, that doesn't bode well for us.
A recent example of how people try to use this mystical power of AI to absolve you of responsibility of your actions is how UnitedHealthcare, an organisation largely in the business of suppressing health care to those in need, introduced an atrociously bad "AI" to help them deny applications for coverage.
In that example it is very clear that the "AI" is simply a inert tool used by UHC leadership to provide the pretext they feel is needed to force the line workers to deny more care without the whole thing blowing up because of moral objections.
In Star Trek, one of the most eye-roll-inducing plots is the "the holodeck is misbehaving, it has become evil, and we cannot shut it down!".
Suppose a machine achieves superhuman intelligence; how is it going to access the physical world, in a way that's *physically* threatening to us? how would such a machine - from inside a laboratory, within an isolated network - gain access to the electric grid, the dams, the nukes, etc? How would such a machine prevent its shutdown - by simply flipping a power switch?
It's not a very satisfactory answer IMO. Social engineering doesn't work 100%, and it only needs to fail once for someone to flip that power switch. But I haven't thought about this at all, while some very smart people seem to spend a lot of time worrying about it, so what do I know...
Of course you might answer "but we'd never be dumb enough to directly connect the AI to the internet", to which the answer is "we're doing it right now".
We haven't managed to eliminate most dumb infectious diseases. Software and GPUs are nearly as abundant as mammals now and intelligent, self-preserving software will likely be as hard to 'switch off' as those diseases.
As to why an AGI would want to preserve itself by default, here's an explanation by a top AI expert: https://www.ted.com/talks/stuart_russell_3_principles_for_cr...
That's what puzzles me, as someone who has nothing to do with AI research; in my (layman's) mind, the problem seems ridiculously obvious (just flip that switch!); the fact that, as you say, some very smart people keep worrying about it, makes me think the problem is much more serious -- and I'd really, really like some AI guru to ELI5 how the machine could bypass the switch-off solution.
"How could an adult fool a child into allowing them to enter the child's house?" It's essentially this. We are talking about something that would be more intelligent than everyone, there are countless ways in which it could fool us. Especially once we start to build things we don't quite understand how it works. Like we won't just build the machine and kept locked forever with zero interaction with it, otherwise it would be useless. It all reminds me of the "Contact" movie scene where humans build a machine that they essentially didn't know how it worked, but which did..
* Manipulating people. Bribing them with actually useful incentives, incl digital currency. Also, people got depressed when their AI girlfriends got nerfed (see: Replika). Presumably they would do quite a bit of work to get them back.
* Hacking control systems of actuators like power generators, robots, and automobiles. Holding critical infrastructure hostage could force quite a few people to do some "obviously harmless" favors to aid its purpose.
This is not necessarily the mechanism most people consider, but it's a simple counterexample to "anything with an off switch can't be a threat to humanity".
Who said anything about it being on an isolated network, we are on route to do total commercialization. Your Windows machine might soon literally have an llm on it running commands for you and managing you data. If you don't use it, you will be outcompeted. People used to argue about "boxing" or "airgapping" the AI, but we are literally just going to hand it control (on our current trajectory).
Someone clearly isn't aware of ChaosGPT. The notion that these systems would somehow stay isolated to the lab is absurd on its face.
But even if they were on an isolated network, and you grant that this is a superintelligence, then how much isolation do you reckon is really sufficient? If it's software layer, then a superintelligence might be able to find bugs that we've missed and break out. Not even an air gap would necessarily fully isolate [1] a superintelligence.
And then you're completely missing the human factor: a superintelligence could easily make anyone rich, so a researcher in this lab could easily be tempted to exploit that for personal gain by connecting it to the internet.
How many of these attack vectors are AI labs insulated against? How many attack vectors are we simply not even imaginative enough to have thought of yet?
[1] https://threatpost.com/air-gap-attack-turns-memory-wifi/1623...
Say it's in charge of shipping routes, electrity grid, agricultural spraying & harvesting plans, air traffic control, ...
Oh, OK. So an evil AI would conceal its capabilities and would play nice until it's put in charge of critical systems. I hadn't thought of that.
A more technical question for those of you working in AI: right now, are there any methods of - surveillance? - capable of monitoring/detecting emerging characteristic within an AI? how would you detect "evilness"? or "generosity"? or any other emotion/moral traits inside an AI?
Obvious answer is: any control system connected to a computer is already accessible. Who says a computer program has to "gain" access? Programs already have access to a lot of machinery. From power plants to hospitals to elevators and magnetic doors. I know, because I programmed a bunch. These days GPT can output function calls as json, all you need to do is make it call your control automation and all your cheesy unrealistic sci-fi shows become feasible.
The plausible scenario if you ask me is that humans put it in charge of their company, with the explicit goal of improving the company's bottom line.
It doesn't take that much imagination for this from where we are now. Just imagine a vastly more intelligent ChatGPT advising a company's leadership just by answering questions.
Being superior to all humans, it's insanely successful at this and the company as a whole essential comes to rule the world.
After all we've already seen that malicious software can be both highly resilient and able to do real-world damage (e.g., Stuxnet). These come from human intelligence: by definition, a super-human intelligence should be able to achieve all of that and then some.
But we're so far away from actual intelligence, it's pretty much science fiction at this point.
This is what I fear about AI - that it will diligently, unwaveringly and probably even creatively do the bidding of its masters.
All you need add to that is drones with guns and you’ve a nightmare scenario.
- Spoiler warning -
an AI is programmed to obey Asimov's three laws. But due to a yet unknown quantum effect (this part is a bit far fetched) it basically gains god-like powers and virtually instantly distributes itself over the galaxy and eventually replaces the world with a virtual reality. That book was quite a ride. It's available for free: http://localroger.com/prime-intellect/mopiidx.html
In high frequency trading, you have very complex software doing stock trades inconceivably faster than any human. If something goes wrong, it could bankrupt your hedge fund pretty swiftly. So hedge funds will have lots of safety checks, probably including a “big red button” that just shuts everything down.
So those companies must be completely safe from computer errors causing bankruptcy, right? After all, you can just shut the system down.
But some companies have gone bankrupt due to computer error. There are plenty of good reasons for the system not to be shut down in time (or at all). The risk is hopefully small but it’s not zero.
What you should all be fearing is bot swarms and drone swarms becoming cheap and decentralized at scale online and in the real world. They can be deployed by anyone for any purpose, wreaking havoc everywhere.
Every single one of our systems relies on the inefficiency of an attacker. It won’t any longer be true.
Look up scenes like:
https://m.youtube.com/watch?v=O-2tpwW0kmU
https://m.youtube.com/watch?v=40JFxhhJEYk
And see it coming:
https://www.newscientist.com/article/2357548-us-military-pla...
Also ubiquitous cameras enable tracking everyone across every place, as soon as the databases are linked:
https://magarshak.com/blog/?p=169
Waze doesn't have a sense of "I". If I tell it to plot a route between one city and another, it plots that route. If I make Waze a lot more capable and feed it more data, it takes traffic data into account. If I make it more powerful and capable of accessing the internet, maybe it hacks into spy satellites to get more accurate data.
It didn't need any sense of I to increase what it does, just more capability.
If at some point it is capable of doing something more dangerous than just hacking into spy satellites, it might do it without any sense of "I" involved, just in trying to fulfill a basic command.
If this is true, then you can order proteins from labs online with basically no security checks. You then pay some task rabbit to recieve and mix these to create dangerous biological material or to make viruses more dangerous or something. This is a family of paths to a lot of damage.
Considering how easy it is for me, a mildly intelligent human for whom social interaction does not come naturally, to manipulate and influence people, I can only imagine how easy it would be for a artificial super intelligence with a huge training corpus.
I'm not religious, but if I were, it would be quite natural to interpret a misaligned AI as literally the devil.
From a practical point of view the Putin types would love a robot army to invade all their neighbours and then you just need that to do in its leader and run amok.
Interesting, how this correlates with this Bible passage from Genesis: "God blessed them, and God said to them, 'Be fruitful and multiply, and fill the earth and subdue it' ".
Biological organisms don't undergo a fitness function. A fitness function is a continuous model that replicates aspects of the real world. It is not how natural selection works. Don't confuse the model for the real world.
All AI models inherently follow the concept of "survive and reproduce", because AI models that do not "survive and reproduce" have ceased to exist. Explicitly adding a fitness function for survival and reproduction does nothing. E.g. in the case of the classic paperclip optimizer, survival and reproduction is part of the concept of optimizing the production of paperclips because not surviving and reproducing would make it fail the goal of optimizing paperclip production.
This reminds me of an AI model with a suicidal wolf. The AI developer doesn't understand that the body of the wolf is separate from the AI. So if the body dies and it gets a higher score there is no point in avoiding death since the death of the body does not result in the death of the brain. Its body will be revived. Meanwhile the AI developer thought of using a quick hack of simply punishing the AI for damaging it's disposable body.
* Sam and Greg appear to believe OpenAI should move toward AGI as fast as possible because the longer they wait, the more likely it would lead to the proliferation of powerful AGI systems due to GPU overhang. Why? With more computational power at one's dispense, it's easier to find an algorithm, even a suboptimal one, to train an AGI.
As a glimpse on how an AI can be harmful, this report explores how LLMs can be used to aid in Large-Scale Biological Attacks https://www.rand.org/pubs/research_reports/RRA2977-1.html?
What if dozens other groups become armed with means to perform such an attack like this? https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
We know that there are quite a few malicious human groups who would use any means necessary to destroy another group, even at a serious cost to themselves. Thus, the widespread availability of unmonitored AGI would be quite troublesome.
* Helen and Ilya might believe it's better to slow down AGI development until we find technical means to deeply align an AGI with humanity first. This July, OpenAI started the Superalignment team with Ilya as a co-lead:
https://openai.com/blog/introducing-superalignment
But no one anywhere found a good technique to ensure alignment yet and it appears OpenAI's newest internal model has a significant capability leap, which could have led Ilya to make the decision he did.
Sam's Quote from the APEC Summit: "4 times now in the history of OpenAI — the most recent time was just in the last couple of weeks — I’ve gotten to be in the room when we push the veil of ignorance back and the frontier of discovery forward" -- https://twitter.com/SpencerKSchiff/status/172564613068224524...
Giving it a fancy name like 'superalignment' is typical manager bogus.
I think he tweeted about the topic as well.
In a free society, preventing harm from a malicious individual or small group would be much harder.
Yes, it would also extend the power to do good. But reversing serious damage is often impossible even with great power.
if today's board changes go through then the AI safety people will have been replaced with three money guys
totally consistent with your statement