Norbert Wiener was ahead of his time in recognizing the potential danger of emergent intelligent machines. I believe he was even further ahead in recognizing that the first artificial intelligences had already begun to emerge. He was correct in identifying the corporations and bureaus that he called "machines of flesh and blood" as the first intelligent machines.
I remember reading Bostrom's work in 2014 and raving about it to others while no one really understood what I was so interested in. Well, now everyone is talking about this topic. One of my favorite analogies in the book goes something like, imagine a worm wriggling in the ground, it has no conception of the god-like beings that inhabit the world, in cities, having all sorts of goals, doing all sorts of jobs. It literally does not have the brain power to comprehend what is happening.
This seems like a poor metaphor, considering that we can understand what constituent things would make up a superintelligence, even if we don’t understand the whole.
This discussion centers too much on the definitions of words like superintelligent and reminds me a lot of philosophical discussion about omnipotence. Both seem to rely more on defining concepts first and then assuming their existence as a consequence.
Omnipotence is provably impossible: "Could Jesus microwave a burrito so hot that he himself could not eat it?" etc.
Super-intelligence, not so much — there's plenty of examples of above-average humans in many areas, and no reason to think that putting the top expert in each field into one room is impossible, and no reason to think that this configuration cannot be implemented in software with a sufficiently powerful computer.
And that's without the things that machines already beat us at, because super-human chess playing software is easily available, and computers that do arithmetic better than the entire human species even if we were all trained to the level of the current world record holder are cheap enough to be given away free glued to the front of a magazine, so there's no single person who has a particular advantage with those things.
What chess does do, is give an example: if I was playing a game with Kasparov, I would have no idea which move he might make at any given moment, but despite that I'd still expect him to win.
With an AI, I don't even necessarily know what "game" it's playing even if I'm the one who wrote the utility function it's trying to maximise.
> considering that we can understand what constituent things would make up a superintelligence
Can we? What constituent things would make up a superintelligence? Who's to say that our answer to that question is what is actually true in the case of a real superintelligence?
> Both seem to rely more on defining concepts first and then assuming their existence as a consequence.
Unlike religious philosophers like Anselm of Canterbury and Descartes and their ontological argument for the existence of a supreme being merely by imagining it, I don't believe anyone in the study of superintelligence is presupposing that they exist, or even can exist, they only presuppose how one might hypothetically exist.
We understand some of it but who can say we understand the majority of it ? We might be at 0.1% of understanding the reality without be able to state this. Just as a worm surely « understand » some of it to differentiate and process it’s surroundings.
I always thought about it like being a permanent infant. The world is huge, full of colorful things you don't understand, and it'll be like that forever. But it's also a poor metaphor because adults have something that toddlers don't: fear. As a kid, you're an ignorant and curious blank slate; as an adult, you've established expectations and anxieties, so you'd probably be having a much worse time. :D
Imagine a clocked-down Turing computer, whose operating speed us is so slow that we can see its operations. Now imagine the programs which cannot, in principle be run on this computer. You can't because there aren't any.
Now imagine we are that computer.
Humans are capable of abstract thought and reasoning, which is sufficient to understand even the most complex software programs, to such an extent as they are understandable. The comparison to the worm is bogus.
The clocked-down Turing computer you are proposing would have to have an infinite amount of tape to be truly "Turing complete" and capable of running any program we can imagine in principle.
To actually match the worm analogy, imagine a clocked-down Turing machine with a tape length of maybe 100k cells. Now you can definitely imagine programs that can't run on it.
The worm's limitation isn't that it's slow at thinking, it basically has only a handful of neurons compared to a person and there are problems/concepts that are just too large to fit in its brain and it would never be able to imagine even if it could live and think until the end of time.
I like to imagine bacteria as the compute substrate for an immaterial city of digital inhabitants. Fungus are even cooler, with the hypothetical wood wide web. maybe we already are the worms!
We already have "superintelligences" in the world; Nature and other humans treated as a collective is far more powerful than any individual. We manage these risks by not trusting them completely, and restricting their dominance over us. I don't see that we can't adapt to superintelligent machines as long as we don't surrender all decision making to them; the risk comes from the same old, where a group possesses overwhelming power that is then used to regiment and oppress a less powerful group. In which case, possession of AI is far from unique.
Bostrom's accomplishment is to succeed at bamboozling many who felt they were beyond being bamboozled.
Bostrom is a charlatan playing rubes with an intellectual version of three-card monty:
//This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation. It follows that the belief that there is a significant chance that we will one day become posthumans who run ancestor‐simulations is false, unless we are currently living in a simulation. A number of other consequences of this result are also discussed.//
What is a post-human stage of development?
Is human extinction a post-human stage of development?
How did this argument get from post-human to ancestor simulation? It's stipulated in a lemma.
What is a simulation according to Bostrum? Undefined.
What then is an "ancestor" simulation?!
The weaseling on display adds gross insult to intellectual injury:
How unlikely is "extremely unlikely"? What is a "significant number" of ancestor" simulations? What is "evolutionary history"? How did he get from ancestor simulation to computer simulation?
Who else can stake an academic reputation on tenets delivered in the form of afterthoughts?
A theory must contain a promise of making matters simpler. If there's any kind of theory associated with Bostrom's argument, it's that many people can't distinguish a theory from an arbitrary collection of intriguing statements. Bostrom is functioning as an academic instance of Weisenbaum's Eliza: it just circularly echoes its own kooky conjectures in the form of lemmas and an outwardly spiraling discussion of itself.
Bostrom is playing a silly joke on readers. He gives his game away with his ridiculous lemmmas, showing the ace during the shuffle:
"It follows that the belief that there is a significant chance that we will one day become posthumans who run ancestor‐simulations is false"—period.
So, what was the question which was to be begged? Read it and weep. You've been played.
You've got to hand it to a magician / mentalist who gets his audience to believe they are as "worms wiggling in the ground with no conception of god-like beings that inhabit the world".
The most pitiless aspect of this fantasy is that it includes a proposition for what man is unaware of in the form of the overt statement of that which he is unaware: "god-like beings" Could the question of faith be any further begged? It's deranged.
Here we encounter Zizek's fourth quadrant of Rumsfeld's knowns: the "unknown knowns." Ideology. That which you know but avoid or refuse to become aware that you know to keep your faith alive.
There's a rule about the actual unknown, which is that it must be literally about the which is not known!
More generally, as to what is not known, we can only poke at the edge of a lacunae, maybe inferring something about the contours or gradient to mystery, maybe following the edge to refining our adaptation to ourselves. But our work doesn't belong with the unknowable. It belongs with what we can know about ourselves. The unknown will take care of itself.
In intellectual terms, I'm currently interested in the fusion of Asian and Western history, reading James Clavell's Asian Saga now, after watching Shogun recently. David Graeber's books are also on my list once I finish the Saga. I've read Bullshit Jobs and Debt by him but I've heard good things about The Dawn of Everything, particularly how European Enlightenment ideas might have actually been influenced by what they saw from Native Americans.
In terms of projects I'm working on, I'm traveling currently and it's a pain to track how much money I've spent due to needing to convert foreign currencies, so I'm building a simple app for that.
I just want to point out that the paperclip problem was already present in a short story from 1959 by a Soviet sci-fi writer Dneprov - Crabs on the Island
The author states that AI safety is very important, that many experts think it is very important and that even governments consider it to be very important, but there is no mention of why it is important or what "safe" AI even looks like.
Am I that out of the loop that what this concept entails is so obvious that it doesn't require an explanation, or am I overlooking something here?
The idea that most AIs are unsafe to non-AI interests is foundational to the field and typically called instrumental convergence [1]. You can also look up the term "paperclip maximizer" to find some concrete examples of what people fear.
It's unfortunately hard to describe what a safe AI would look like, although many have tried. Similar to mathematics, knowing what the correct equation looks like is a huge advantage in building the proof needed to arrive at it, so this has never bothered me much.
You can see echoes of instrumental convergence in your everyday life if you look hard enough. Most of us have wildly varying goals, but for most of those goals, money is a useful way to achieve them -- at least up to a point. That's convergence. An AI would probably get a lot farther by making a lot of money too, no matter what the goal is.
Where this metaphor breaks down is we human beings often arrive at a natural satiety point with chasing our goals: We can't just surf all day, we eventually want to sleep or eat or go paddle boarding instead. A surfing AI would have no such limiters, and might do such catastrophic things as use its vast wealth to redirect the world's energy supplies to create the biggest Kahuna waves possible to max out its arbitrarily assigned SurfScore.
The article itself is talking about a specific book. "Superintelligence: Paths, Dangers, Strategies" by Nick Bostrom. That book is the seminal work on the subject of AI safety. If you honestly want answers to your questions I recommend reading it. It is written in a very accessible way.
If reading a whole book is out of question then I'm sure you can find many abridged versions of it. In fact the article itself provides some pointers at the very end of it.
> Am I that out of the loop
Maybe? Kinda? That's the point of the article. There has been 10 years since the publication of the book. During that time the topic went from the weird interest of some Oxford philosopher to a mainstream topic discussed widely. 10 years is both a long time and a blink of an eye. Depending on your frame of reference. But it is never too late to get in the loop if you want to.
At the same time I don't think it is fair to expect from every article ever to rehash the basic concepts of the field they are working on.
Many have expressed my sentiments far better than I can, but Superintelligence is quite frankly written in a very tedious way. He says in around 300 pages what should have been an essay.
I also found some of his arguments laughably bad. He mentions that AI might create a world of a handful of trillionaires, but doesn’t seem to see this extreme inequality as an issue or existential threat in and of itself.
> AI is safe if it does not cause extinction of humanity.
I don't think that is true. "AI is not safe if it cause extinction of humanity." is more likely to be true. But that is a necessary requirement but not sufficient.
Just think of a counter example: An AI system which wages war on humanity, wins and then keeps a stable breeding population of humans in abject suffering in a zoo like exhibit. This hypothetical AI did not cause extinction of humanity. Would you consider it safe? I would not.
Yeah, the constant dissonance with AI safety is that every single AI safety problem is already a problem with large corporations not having incentives aligned with the good of people in general. Profit is just another paperclip.
This is one of the most delusional and speculative books I've ever read. The author comes up with elaborate analytical models resting on slippery, loosely-defined terms. Being smart with algebra while totally disconnected from technological grounds. It's the kind of stuff VP execs and Bill Gates like to read, and one of the reasons for the current bubble.
My annoyance is that _anyone_ would write a book on a technical subject who knows absolutely nothing about the subject. LLMs aren't a philosophical concept; they're a software mechanism with myriad constraints and design limitations built in. Understanding their future demands a deep understanding of those mechanisms. So why on earth would an academic who knows zero about engineering, software, or AI techniques have the temerity to write a book suggesting he can see farther into the evolution of LLMs than, say, a carpenter or bricklayer? At least those skills know something about physical mechanisms and engineering constraints. But not Bostrom.
The continued interest in a book of bold uninformed argumentation that's so obviously insubstantial just goes to show how bad humans are at telling the difference between useful knowledge and wild speculation. It's almost as silly as caring whether the prognostications of Rod Brooks (or worse still, Ray Kurzweil) come true. As if guessing right actually meant something...
I've re-skimmed it recently as well, and found it to be extremely zeerusted and needlessly alarmist in retrospect. A lot of it is written from the perspective of "a handful of scientists build brain in a bunker a la Manhattan project" that is so far from our actual reality that 90% of the concerns don't even apply.
Exponential runaway turned out to not be a thing at all, progress is slow (on the order of years), competitors are aplenty, alignment is easy, everything is more or less done in the open with papers being published every day. We're basically living out the absolute best possible option out of all the ones outlined in the book.
Looks like the real-world risks of AI are, predictably, AI being used to avoid responsibility/liability/regulation or plainly copyright-laundering (which likewise predictably is only a temporary loophole until laws catch up) and companies like Google reversing all progress they made in reducing their emissions by doubling down on resource-intense AI.
"Avoiding regulation" as a Service of course has a huge market potential for as long as it works, just like it did for crypto and the gig economy. But it is by definition a bubble because it will deflate as soon as the regulations are fixed. GenAI might have an eventual use but it will in all likelihood look nothing like what it is used for at the moment.
And yeah, you could complain that what I said mostly applies to GenAI and LLMs but that's where the current hype is. Nobody talks about expert systems because they've been around for decades and simply work while being very limited and "unsexy" because they don't claim to give us AGI.
The problem starts with talking about "AGI" and LLMs/GenAI in the same breath. LLMs are not and can not be AGI. They are impressive, but they are glorified autocomplete. When ChatGPT lets you "correct" it, it doesn't backtrack, it takes your response into consideration along with what it said before and generates what its model suggests could come next in the conversation. It's more similar to a Markov chain than to an expert system.
Im going to make play devils advocate here, but I think LLMs are the closest we can get to AGI, because AGI is a silly concept. Literally nobody on earth has “General Intelligence,” nobody is generally capable in all things, so why do we expect software to be?
Still, the average cutting edge LLM does a hell of a lot better at a great many things than a great many humans. I know, it’s just computer, but what is the average skill level? We just keep moving the goal posts.
The author claims that we are "between third and fifth point" in the following list:
>i Safety alarmists are proved wrong
>ii Clear relationship between AI intelligence and safety/reliability
>iii Large and growing industries with vested interests in robotics and machine intelligence.
>iv A promising new technique in artificial intelligence, which is tremendously exciting to those who have participated in or followed the research.
>v The enactment of some safety rituals, whatever helps demonstrate that the participants are ethical and responsible (but nothing that significantly impedes the forward charge).
>vi A careful evaluation of seed AI in a sandbox environment, showing that it is behaving cooperatively and showing good judgment.
Have we really gone past the first point? After decades of R&D, driverless cars are still not as safe as humans in all conditions. We have yet to see the impact of generative AI on the intellectual development of software engineers, or to what extent it will exacerbate the "enshittification" of software. There's compelling evidence that nation states are trusting AI to identify "likely" terrorists who are then indiscriminately bombed.
The abridged summary here elides that 1 is a history of claims of intolerable harm being proved wrong, not that every claim has already been proved wrong. In this frame that too many people kept raising alarms equivalent to "cars with driving assistance will cause a bloodbath" which then come to pass, not that there are no further safety alarmist claims left about what could be coming next as the technology changes.
Keeping it focused on AI every release of a text, image, and voice generator has come with PR, delays, news articles, and discussion about how it's dangerous and we need to hold it back. 3 months after they release politics hasn't collapsed from a 10 fold increase in fake news, discussion boards online are still as (un)usable as they were before, art is still a thing people do, and so on. That doesn't mean there are no valid safety concerns just that the alarmist track record isn't particularly compelling to most while the value of the tools continues to grow.
I think it will always depend on who you ask, and if they're arguing in bad faith:
"Sure, the sentry bot can mistakenly shoot and kill its own owner and/or family, but only if they're carrying a stapler. Like, who even uses a stapler in this day and age?"
I’ve been working on this issue for a while and the conclusion I have come to is:
We’re not going to see actual movement on managing AI risk until there is the equivalent of a Hiroshima/three mile island/chernobyl from a self-improving system that has no human in the loop.
Not enough people actually believe ASI is possible and harmful, to create a movement that will stop the people who are pursuing it who don’t care or don’t believe its going to be harmful.
It would have been impossible to have a nuclear weapons ban prior to World War II because 1. Almost nobody knew about it 2. Nobody would have actually believed it could be that bad
The question you should ask is, if someone does make it, on any timeline is there any possible counter at that point?
It seems like your nuclear analogy might have an answer to the question. Take that theory that it could have been possible the nuclear chain reaction ignited the atmosphere.
It seems like the worst-case predictions about AI are at that scale. There is also the possibility that AI causes problems on a much smaller scale. A country’s weapons system goes terribly wrong. A single company’s business goes berserk. Stock trading bot at a large enough bank/fund causes a recession. Stuff like that seems rather likely to happen as systems are adopted before they’re “ready” and before they’re anything close to AGI.
The book “weapons of math destruction” is full of examples of AI fucking up badly and ruining people’s lives today. It’s just that the damage is more diffuse than Hiroshima and nowhere NEAR as gruesome and specially located and sudden.
Perfect is the enemy of good, so why vote for a lesser good?
Humans are so existentially biased and self-centred!
And they are always forgetting that they wouldn't even be there if others hadn't made room for them.From the Great Oxygen to the K–Pg extinction event.
Be generous!
"Man is something that shall be overcome. What have you done to overcome him?"
It's going to happen biologically before it happens in silicon, anyway. And the biological venue could very well be humans (genetically modified). So I quite literally agree. :)
Readit News
https://en.wikipedia.org/wiki/Possible_Minds
https://libcom.org/history/father-cybernetics-norbert-wiener...
Fortunately, none of these qualify as paperclip maximizers
https://en.wikipedia.org/wiki/Darwin_among_the_Machines
Now imagine we are the worm.
This discussion centers too much on the definitions of words like superintelligent and reminds me a lot of philosophical discussion about omnipotence. Both seem to rely more on defining concepts first and then assuming their existence as a consequence.
Super-intelligence, not so much — there's plenty of examples of above-average humans in many areas, and no reason to think that putting the top expert in each field into one room is impossible, and no reason to think that this configuration cannot be implemented in software with a sufficiently powerful computer.
And that's without the things that machines already beat us at, because super-human chess playing software is easily available, and computers that do arithmetic better than the entire human species even if we were all trained to the level of the current world record holder are cheap enough to be given away free glued to the front of a magazine, so there's no single person who has a particular advantage with those things.
What chess does do, is give an example: if I was playing a game with Kasparov, I would have no idea which move he might make at any given moment, but despite that I'd still expect him to win.
With an AI, I don't even necessarily know what "game" it's playing even if I'm the one who wrote the utility function it's trying to maximise.
Can we? What constituent things would make up a superintelligence? Who's to say that our answer to that question is what is actually true in the case of a real superintelligence?
> Both seem to rely more on defining concepts first and then assuming their existence as a consequence.
Unlike religious philosophers like Anselm of Canterbury and Descartes and their ontological argument for the existence of a supreme being merely by imagining it, I don't believe anyone in the study of superintelligence is presupposing that they exist, or even can exist, they only presuppose how one might hypothetically exist.
Now imagine we are that computer.
Humans are capable of abstract thought and reasoning, which is sufficient to understand even the most complex software programs, to such an extent as they are understandable. The comparison to the worm is bogus.
To actually match the worm analogy, imagine a clocked-down Turing machine with a tape length of maybe 100k cells. Now you can definitely imagine programs that can't run on it.
The worm's limitation isn't that it's slow at thinking, it basically has only a handful of neurons compared to a person and there are problems/concepts that are just too large to fit in its brain and it would never be able to imagine even if it could live and think until the end of time.
Bostrom is a charlatan playing rubes with an intellectual version of three-card monty:
//This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation. It follows that the belief that there is a significant chance that we will one day become posthumans who run ancestor‐simulations is false, unless we are currently living in a simulation. A number of other consequences of this result are also discussed.//
What is a post-human stage of development?
Is human extinction a post-human stage of development?
How did this argument get from post-human to ancestor simulation? It's stipulated in a lemma.
What is a simulation according to Bostrum? Undefined.
What then is an "ancestor" simulation?!
The weaseling on display adds gross insult to intellectual injury:
How unlikely is "extremely unlikely"? What is a "significant number" of ancestor" simulations? What is "evolutionary history"? How did he get from ancestor simulation to computer simulation?
Who else can stake an academic reputation on tenets delivered in the form of afterthoughts?
A theory must contain a promise of making matters simpler. If there's any kind of theory associated with Bostrom's argument, it's that many people can't distinguish a theory from an arbitrary collection of intriguing statements. Bostrom is functioning as an academic instance of Weisenbaum's Eliza: it just circularly echoes its own kooky conjectures in the form of lemmas and an outwardly spiraling discussion of itself.
Bostrom is playing a silly joke on readers. He gives his game away with his ridiculous lemmmas, showing the ace during the shuffle:
"It follows that the belief that there is a significant chance that we will one day become posthumans who run ancestor‐simulations is false"—period.
So, what was the question which was to be begged? Read it and weep. You've been played.
You've got to hand it to a magician / mentalist who gets his audience to believe they are as "worms wiggling in the ground with no conception of god-like beings that inhabit the world".
The most pitiless aspect of this fantasy is that it includes a proposition for what man is unaware of in the form of the overt statement of that which he is unaware: "god-like beings" Could the question of faith be any further begged? It's deranged.
Here we encounter Zizek's fourth quadrant of Rumsfeld's knowns: the "unknown knowns." Ideology. That which you know but avoid or refuse to become aware that you know to keep your faith alive.
There's a rule about the actual unknown, which is that it must be literally about the which is not known!
More generally, as to what is not known, we can only poke at the edge of a lacunae, maybe inferring something about the contours or gradient to mystery, maybe following the edge to refining our adaptation to ourselves. But our work doesn't belong with the unknowable. It belongs with what we can know about ourselves. The unknown will take care of itself.
In terms of projects I'm working on, I'm traveling currently and it's a pain to track how much money I've spent due to needing to convert foreign currencies, so I'm building a simple app for that.
https://archive.org/stream/AnatolyDneprovCrabsOnTheIsland/An...
I am sure there are even earlier examples - but the above is a nice short read.
[1]: https://en.m.wikipedia.org/wiki/Instrumental_convergence
It's unfortunately hard to describe what a safe AI would look like, although many have tried. Similar to mathematics, knowing what the correct equation looks like is a huge advantage in building the proof needed to arrive at it, so this has never bothered me much.
You can see echoes of instrumental convergence in your everyday life if you look hard enough. Most of us have wildly varying goals, but for most of those goals, money is a useful way to achieve them -- at least up to a point. That's convergence. An AI would probably get a lot farther by making a lot of money too, no matter what the goal is.
Where this metaphor breaks down is we human beings often arrive at a natural satiety point with chasing our goals: We can't just surf all day, we eventually want to sleep or eat or go paddle boarding instead. A surfing AI would have no such limiters, and might do such catastrophic things as use its vast wealth to redirect the world's energy supplies to create the biggest Kahuna waves possible to max out its arbitrarily assigned SurfScore.
If reading a whole book is out of question then I'm sure you can find many abridged versions of it. In fact the article itself provides some pointers at the very end of it.
> Am I that out of the loop
Maybe? Kinda? That's the point of the article. There has been 10 years since the publication of the book. During that time the topic went from the weird interest of some Oxford philosopher to a mainstream topic discussed widely. 10 years is both a long time and a blink of an eye. Depending on your frame of reference. But it is never too late to get in the loop if you want to.
At the same time I don't think it is fair to expect from every article ever to rehash the basic concepts of the field they are working on.
Many have expressed my sentiments far better than I can, but Superintelligence is quite frankly written in a very tedious way. He says in around 300 pages what should have been an essay.
I also found some of his arguments laughably bad. He mentions that AI might create a world of a handful of trillionaires, but doesn’t seem to see this extreme inequality as an issue or existential threat in and of itself.
The article does link to "Statement on AI Risk", at https://www.safe.ai/work/statement-on-ai-risk
It is very short, so here is full quote.
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
I don't think that is true. "AI is not safe if it cause extinction of humanity." is more likely to be true. But that is a necessary requirement but not sufficient.
Just think of a counter example: An AI system which wages war on humanity, wins and then keeps a stable breeding population of humans in abject suffering in a zoo like exhibit. This hypothetical AI did not cause extinction of humanity. Would you consider it safe? I would not.
Whether this concept is actionable is another matter.
The continued interest in a book of bold uninformed argumentation that's so obviously insubstantial just goes to show how bad humans are at telling the difference between useful knowledge and wild speculation. It's almost as silly as caring whether the prognostications of Rod Brooks (or worse still, Ray Kurzweil) come true. As if guessing right actually meant something...
There aren’t any non-philosophical computer science concepts.
Exponential runaway turned out to not be a thing at all, progress is slow (on the order of years), competitors are aplenty, alignment is easy, everything is more or less done in the open with papers being published every day. We're basically living out the absolute best possible option out of all the ones outlined in the book.
"Avoiding regulation" as a Service of course has a huge market potential for as long as it works, just like it did for crypto and the gig economy. But it is by definition a bubble because it will deflate as soon as the regulations are fixed. GenAI might have an eventual use but it will in all likelihood look nothing like what it is used for at the moment.
And yeah, you could complain that what I said mostly applies to GenAI and LLMs but that's where the current hype is. Nobody talks about expert systems because they've been around for decades and simply work while being very limited and "unsexy" because they don't claim to give us AGI.
The UI doesn't let you do that*, the underlying model does. (And so would an actual Markov chain).
* EDIT: not in the middle of a response at least, but it does allow you to backtrack to a previous message and go again from there.
Still, the average cutting edge LLM does a hell of a lot better at a great many things than a great many humans. I know, it’s just computer, but what is the average skill level? We just keep moving the goal posts.
>i Safety alarmists are proved wrong
>ii Clear relationship between AI intelligence and safety/reliability
>iii Large and growing industries with vested interests in robotics and machine intelligence.
>iv A promising new technique in artificial intelligence, which is tremendously exciting to those who have participated in or followed the research.
>v The enactment of some safety rituals, whatever helps demonstrate that the participants are ethical and responsible (but nothing that significantly impedes the forward charge).
>vi A careful evaluation of seed AI in a sandbox environment, showing that it is behaving cooperatively and showing good judgment.
Have we really gone past the first point? After decades of R&D, driverless cars are still not as safe as humans in all conditions. We have yet to see the impact of generative AI on the intellectual development of software engineers, or to what extent it will exacerbate the "enshittification" of software. There's compelling evidence that nation states are trusting AI to identify "likely" terrorists who are then indiscriminately bombed.
Keeping it focused on AI every release of a text, image, and voice generator has come with PR, delays, news articles, and discussion about how it's dangerous and we need to hold it back. 3 months after they release politics hasn't collapsed from a 10 fold increase in fake news, discussion boards online are still as (un)usable as they were before, art is still a thing people do, and so on. That doesn't mean there are no valid safety concerns just that the alarmist track record isn't particularly compelling to most while the value of the tools continues to grow.
I think it will always depend on who you ask, and if they're arguing in bad faith:
"Sure, the sentry bot can mistakenly shoot and kill its own owner and/or family, but only if they're carrying a stapler. Like, who even uses a stapler in this day and age?"
We’re not going to see actual movement on managing AI risk until there is the equivalent of a Hiroshima/three mile island/chernobyl from a self-improving system that has no human in the loop.
Not enough people actually believe ASI is possible and harmful, to create a movement that will stop the people who are pursuing it who don’t care or don’t believe its going to be harmful.
It would have been impossible to have a nuclear weapons ban prior to World War II because 1. Almost nobody knew about it 2. Nobody would have actually believed it could be that bad
The question you should ask is, if someone does make it, on any timeline is there any possible counter at that point?
It seems like the worst-case predictions about AI are at that scale. There is also the possibility that AI causes problems on a much smaller scale. A country’s weapons system goes terribly wrong. A single company’s business goes berserk. Stock trading bot at a large enough bank/fund causes a recession. Stuff like that seems rather likely to happen as systems are adopted before they’re “ready” and before they’re anything close to AGI.
“There’s No Fire Alarm For AGI” https://intelligence.org/2017/10/13/fire-alarm/
Humans are so existentially biased and self-centred!
And they are always forgetting that they wouldn't even be there if others hadn't made room for them.From the Great Oxygen to the K–Pg extinction event.
Be generous!
"Man is something that shall be overcome. What have you done to overcome him?"
Friedrich Nietzsche