One of my formative impressions of AI came from the depiction of the Colligatarch from Alan Dean Foster's The I Inside.
The AI in the book is almost feels like it is the main message masquerading as a subplot.
Asimov knew the risks, and I had assumed until fairly recently that the lessons and explorations that he had imparted into the Robot books had provided a level of cultural knowledge of what we were about to face. Perhaps the movie of I Robot was a warning of how much the signal had decayed.
I worry that we are sociologically unprepared, and sometimes it seems wilfully so.
People discussed this potential in great detail decades ago, Indeed the Sagan reference at the start of this post points to one of the significant contributors to the conversation, but it seems by the time it started happening, everyone had forgotten.
People are talking in terms of who to blame, what will be taken from me, and inevitability.
Any talk of a future we might want dismissed as idealistic or hype. Any depiction of a utopian future is met with derision far too often. Even worse the depiction can be warped to an evil caricature of "What they really meant".
How do we know what course to take if we can't talk about where we want to end up?
I think people broadly feel like all of this is happening inevitably or being done by others. The alignment people struggle to get their version of AI to market first - the techies worry about being left behind. No one ends up being in a position to steer things or have any influence over the future in the race to keep up.
So what can you and I do? I know in my gut that imagining an ideal outcome won't change what actually happens, and neither will criticizing it really.
In the large, ideas can have a massive influence on what happens. This inevitability that you're expressing is itself one of those ideas.
Shifts of dominant ideas can only come about through discussions. And sure, individuals can't control what happens. That's unrealistic in a world of billions. But each of us is invariably putting a little but of pressure in some direction. Ironically, you are doing that with your comment even while expressing the supposed futility of it. And overall, all these little pressures do add up.
Engage respectfully, Try and see other points of view, Try and express your point of view. I decided some time ago that I would attempt to continue conversations on here to try and at least get people to understand that other points of view could be held by rational people. It has certainly cost me Karma, but I hope there has been a small amount of influence. Quite often people do not change their minds by losing arguments, but by seeing other points of view and then given time to reflect.
>I know in my gut that imagining an ideal outcome won't change what actually happens
You might find that saying what you would like to see doesn't get heard, but you just have to remember that you can get anything you want at Alice's Restaurant (if that is not too oblique of a reference)
Talk about what you would like to see, If others would like to see that too, then they might join you.
I think most people working in AI are doing so in good faith and are doing what they think is best. There are plenty of voices telling them how not to it, many of those voices are contradictory. The instances of people saying what to do instead are much fewer.
If you declare that events are inevitable then you have lost. If you characterise Sam Altman as a sociopath playing the long game of hiding in research for years just waiting to pounce on the AI technology that nobody thought was imminent, then you have created a world in you mind where you cannot win. By imagining an adversary without morality it's easy to abdicate the responsibility of changing their mind, you can simply declare it can't be done. Once again choosing inevitability.
Perhaps try and imagine the world you want and just try and push a tiny fraction towards that world. If you are stuck in a seaside cave and the ocean is coming in, instead of pushing the ocean back, look to see if there is an exit at the other end, maybe there isn't one, but at least go looking for it, because if there is, that's how you find it.
As an AI researcher who regularly attend NeurIPS, ICLR, ICML, AAAI (where I am shitposting from). The median AI researcher does not read science fiction, cyberpunk, etc. Most of them haven't read a proper book in over a decade.
Don't expect anyone building these systems to know what Bladerunner is, or "I have no mouth and I must scream" or any other great literature about the exact thing they are working on!
My interpretation is that Asimov assumed that humans would require understanding at the deepest levels of artificial intelligence before it could be created. He built the robot concepts rooted in the mechanical world rather than the world of the integrated circuit.
He never imagined, I suppose, that we would have the computing power necessary to just YOLO-dump the sum of all human knowledge into a few math problems and get really smart sounding responses generated in return.
The risks can be generalized well enough. Man’s hubris is its downfall etc etc.
But the specific issues we are dealing with have little to do with us feeling safe and protected behind some immutable rules that are built into the system.
When Asimov wrote those works there was optimism that Symbolic artificial intelligence would provide the answers.
>But the specific issues we are dealing with have little to do with us feeling safe and protected behind some immutable rules that are built into the system
If your interpretation of the Robot books was that was suggesting a few immutable rules would make us safe and protected, you may have missed the primary message. The overarching theme was an exploration of what those laws could do, and how they may not necessarily correlate with what we want or even perceive as safe and protected. If anything the rules represented a starting point and the books were presenting a challenge to come up with something better.
Anthropic's work on autoencoding activations down to measurable semantic points might prove a step towards that something better. The fact that they can do manipulations based upon those semantic points does suggest something akin to the laws of robotics might be possible.
When it comes to alignment, the way many describe it, it is simply impossible because humans themselves are not aligned. Picking a median, mean, or lowest common denominator of human alignment would be a choice that people probably cannot agree. We are unaligned on even how we could compromise.
In reality, if you have control over what AI does there are only two options.
1. We can make AI do what some people say,
2. We can make them do what they want (assuming we can make them want)
If we make them do what some people, that hands the power to those who have that say.
I think there will come a time when an AI will perceive people doing something wrong, that most people do not think is wrong, and the AI will be the one that is right. Do we want it to intervene or not? Are we instead happy with a nation developing superintelligence that is subservient to the wishes of say, Vladimir Putin.
We've had many decades of technology since Asimov started writing about robots, and we've seen almost all of it used to make the day-to-day experience of the average worker-bee worse. More tracking. More work after hours. More demands to do more with less. Fewer other humans to help you with those things.
We aren't working 4 hour days because we no longer have to spend half the day waiting on things that were slower pre-internet. We're just supposed to deliver more, and oh, work more hours too since now you've always got your work with you.
Any discussion of today's AI firms has to start from the position of these companies being controlled by people deeply rooted in, and invested in, those systems and the negative application of that technology towards "working for a living" to date.
2. Make the case that this is a preferable state. Make people want it.
3. Make the case that it is sustainable once achieved.
4. Identify specific differences between the preferred destination and where we are now.
5. Avoiding short term and temporary effects, work towards changing the differences to what the destination has. Even if that is only proclaiming that these changes are what you want
6. Show how those changes make us closer to the destination that people want.
Some of these are hard problems, I don't think any are intractable. I think they don't get done because they are hard, and opposing something is easier. Rather than building something you want, you can knock down something you don't like. Sure, that might get you closer to your desired state if you consider nothingness to be better than undesired, but without building you will never get there.
If you want everyone to live in a castle, build a castle and invite everybody over. If you start by destroying huts you will just be making adversaries. The converse is true also, if you want everyone to live in huts, build more huts and invite everyone over. If they don't come it's because you haven't made the case that it is a preferable state. Knocking down the castle is not going to convince them of that.
People can't even have a conversation about any kind of societal issues these days without pointing at the other political tribe and casting aspersions about "what they really meant" instead of engaging with what's actually being said.
Forgetting that if you really can hear a dogwhistle, you're also a dog.
Dario's essay carefully avoids its own conclusion. He argues that AI will democratize mass casualty weapons (biology especially), that human coordination at civilizational scale is impossible, and that human-run surveillance states inevitably corrupt. But he stops short of the obvious synthesis: the only survivable path is an AI-administered panopticon.
That sounds dystopian until you think it through:
- The panopticon is coming regardless. The question is who runs it.
- Human-run = corruption, abuse, "rules for thee but not for me."
- AI-run = potentially incorruptible, no ego, no blackmail, no bullshit.
- An AI doesn't need to watch you in any meaningful sense. It processes, flags genuine threats, and ignores everything else. No human ever sees your data.
- Crucially: it watches the powerful too. Politicians, corporations, billionaires finally become actually accountable.
This is the Helios ending from Deus Ex, and it's the Culture series' premise. Benevolent AI sovereignty isn't necessarily dystopia, and it might be the only path to something like Star Trek.
The reason we can't talk about this is that it's unspeakable from inside the system. Dario can't say it (he's an AI company CEO.) Politicians can't say it because it sounds insanely radical. So the discourse stays stuck on half-measures that everyone knows won't work.
I honestly believe this might be the future to work toward, because the alternatives are basically hell.
Where we want to end up? Normies are still talking about the upcoming AI bubble pop in terms of tech basically reverting to 2022. It's wishful thinking all the way down.
Reverting to a world without deployed AI is in fact where "normies", that is most people without capital, want to end up.
The current AI promise for them goes something like: "Oops this chittering machine will soon be able to do all you're good at and derive meaning from. But hey, at least you will end up homeless and part of a permanent underclass."
And the people building it are (rightfully) worried about it killing humanity. So why do we have to continue on this course again? An advanced society would at this point decide to pause what they are doing and reconsider.
Some people say that human jobs will move to the physical world, which avoids the whole category of “cognitive labor” where AI is progressing so rapidly. I am not sure how safe this is, either. A lot of physical labor is already being done by machines (e.g., manufacturing) or will soon be done by machines (e.g., driving). Also, sufficiently powerful AI will be able to accelerate the development of robots, and then control those robots in the physical world.
I would like to believe that we're about to see a rapid proliferation of useful robots, but progress has been much slower with the physical world than with information-based tasks.
After the DARPA Urban Challenge of 2007, I thought that massive job losses from robotic car and truck drivers were only 5-8 years away. But in 2026 in the US only Waymo has highly autonomous driving systems, in only a few markets. Most embodied tasks don't even have that modest level of demonstrated capability.
I actually worry that legislators -- people with white collar jobs -- will overestimate the near-term capabilities of AI to handle jobs in general, and prematurely build solutions for a "world without work" that will be slow to arrive. (Like starting UBI too early instead of boosting job retraining, leaving health care systems understaffed for hands-on work.)
> But in 2026 in the US only Waymo has highly autonomous driving systems, in only a few markets
10 years ago I predicted that the uptake of autonomous vehicles would be slow but that it would be because of labor protections. While those have had some impact, that isn't really the issue: it's that the cars just don't quite work well enough yet and that last ~20% of function turns out to be both incredibly difficult and incredibly important.
One thing that I've not quite been able to sort of get my head around about the whole AI and future of work thing ss the view around work in the physical world being safe. I don't particularly buy the rationale and not from the position of robots are going to do the work. I don't know much about robots really but from what I've seen from the more viral stuff that breaks through to mainstream internet from time to time, it still feels that we're some way out.
But that feels like the least of the worries to me. There seems to be an implicit assumption that those physical lines of work don't get eroded by the higher proportion of able bodied people who are suddenly unemployable. Yes there is some training required etc. but the barriers to entry aren't so high that in the shortish to medium term you don’t see more people gravitating to those industries and competing wages further down to not make then sustainable employment long term. I'd even think that having LLMs that can recognise photos or understand fuzzily explain questions about some blue collar skills many have forgotten actually reduces the barrier even more
I don't think we have much to worry about in terms of economic disruption. At this point it seems pretty clear that LLMs are having a major impact on how software is built, but for almost every other industry the practical effects are mostly incremental.
Even in the software world, the effect of being able to build software a lot faster isn't really leading to a fundamentally different software landscape. Yes, you can now pump out a month's worth of CRUD in a couple days, but ultimately it's just the same CRUD, and there's no reason to expect that this will change because of LLMs.
Of course, creative people with innovative ideas will be able to achieve more, a talented engineer will be able to embark on a project that they didn't have the time to build before, and that will likely lead to some kind of software surplus that the economy feels on the margins, but in practical terms the economy will continue to chug along at a sustained pace that's mostly inline with e.g. economic projections from 10 years ago.
> At this point it seems pretty clear that LLMs are having a major impact on how software is built, but for almost every other industry the practical effects are mostly incremental.
Even just a year ago, most people thought the practical effects in software engineering were incremental too. It took another generation of models and tooling to get to the point where it could start having a large impact.
What makes you think the same will not happen in other knowledge-based fields after another iteration or two?
> most people thought the practical effects in software engineering were incremental too
Hum... Are you saying it's having clear positive (never mind "transformative") impact somewhere? Can you point any place we can see observable clear positive impact?
Software is more amenable to LLMs because there is a rich source of highly relevant training data that corresponds directly to the building blocks of software, and the "correctness" of software is quasi-self-verifiable. This isn't true for pretty much anything else.
Agreed. I also believe the impact on producing software is also over-hyped and in the long term there will be a pull-back in the usage of the tools as the negative effects are figured out.
The unfortunate truth (for Amodei) is you cant automate true creativity and nor standardise taste. Try as they might.
> I don't think we have much to worry about in terms of economic disruption. At this point it seems pretty clear that LLMs are having a major impact on how software is built, but for almost every other industry the practical effects are mostly incremental.
You clearly didn't read the post. He is talking about AI that is smarter than any human, not today's LLMs. The fact that powerful AI doesn't exist yet doesn't mean there is nothing to worry about.
This kind of petty remark is like a reverse em dash. Greetings fellow human.
Anyway, I did read it. The author's description of a future AI is basically just a more advanced version of LLMs
> By “powerful AI,” I have in mind an AI model—likely similar to today’s LLMs in form, though it might be based on a different architecture, might involve several interacting models, and might be trained differently—with the following properties:
They then go on to list several properties that meet their definition, but what I'm trying to explain in my comment is that I don't accept them all at face value. I think it's fair to critique from that perspective since the author explicitly modeled their future based on today's LLMs, unlike many AI essays that skip straight to the super intelligence meme as their premise.
It's interesting just how many opinions Amodei shares with AI 2027's authors despite coming from a pretty different context.
- Prediction of exponential AI research feedback loops (AI coding speeding up AI R&D) which Amodei says is already starting today
- AI being a race between democracies and autocracies with winner-takes-all dynamics, with compute being crucial in this race and global slowdown being infeasible
- Mention of bioweapons and mirror life in particular being a big concern
- The belief that AI takeoff would be fast and broad enough to cause irreplaceable job losses rather than being a repeat of past disruptions (although this essay seems to be markedly more pessimistic than AI 2027 with regard to inequality after said job losses)
- Powerful AI in next few years, perhaps as early as 2027
I wonder if either work influenced the other in any way or is this just a case of "great minds think alike"?
It's because few realize how downstream most of this AI industry is of Thiel, Eliezer Yudkowsky and LessWrong.com.
Early "rationalist" community was concerned with AI in this way 20 years ago. Eliezer inspired and introduced the founders of Google DeepMind to Peter Thiel to get their funding. Altman acknowledged how influential Eliezer was by saying how he is most deserving of a Nobel Peace prize when AGI goes well (by lesswrong / "rationalist" discussion prompting OpenAI). Anthropic was a more X-risk concerned fork of OpenAI. Paul Christiano inventor of RLHF was big lesswrong member. AI 2027 is an ex-OpenAI lesswrong contributor and Scott Alexander, a centerpiece of lesswrong / "rationalism". Dario, Anthropic CEO, sister is married to Holden Karnofsky, a centerpiece of effective altruism, itself a branch of lesswrong / "rationalism". The origin of all this was directionally correct, but there was enough power, $, and "it's inevitable" to temporarily blind smart people for long enough.
It is very weird to wonder, what if they're all wrong. Sam Bankman-Fried was clearly as committed to these ideas, and crashed his company into the ground.
But clearly if out of context someone said something like this:
"Clearly, the most obvious effect will be to greatly increase economic growth. The pace of advances in scientific research, biomedical innovation, manufacturing, supply chains, the efficiency of the financial system, and much more are almost guaranteed to lead to a much faster rate of economic growth. In Machines of Loving Grace, I suggest that a 10–20% sustained annual GDP growth rate may be possible."
I'd say that they were a snake oil salesman. All of my life experience says that there's no good reason to believe Dario's predictions here, but I'm taken in just as much as everyone else.
I really recommend “More Everything Forever” by Adam Becker. The book does a really good job laying out the arguments for AI doom, EA, accelerationism, and affiliated movements, including an interview with Yudkowsky, then debunking them. But it really opened my eyes to how… bizarre? eccentric? unbelievable? this whole industry is. I’ve been in tech for over a decade but don’t live in the bay, and some of the stuff these people believe, or at least say they believe, is truly nuts. I don’t know how else to describe it.
Yeah, it's a pretty blatant cult masquerading as a consensus - but they're all singing from the same hymn sheet in lieu of any actual evidence to support their claims. A lot of it is heavily quasi-religious and falls apart under examination from external perspectives.
We're gonna die, but it's not going to be AI that does it: it'll be the oceans boiling and C3 carbon fixation flatlining that does it.
It used to be a small group of people who mostly just believed that AI is a very important technology overlooked by most. Now, they're vindicated, the importance of AI is widely understood, and the headcount in the industry is up x100. But those people who were on the ground floor are still there, they all know each other, and many keep in touch. And many who entered the field during the boom were those already on the periphery of the same core group.
Which is how you get various researchers and executives who don't see eye to eye anymore but still agree on many of the fundamentals - or even things that appear to an outsider as extreme views. They may have agreed on them back in year 2010.
"AGI is possible, powerful, dangerous" is a fringe view in the public opinion - but in the AI scene, it's the mainstream view. They argue the specifics, not the premise.
I am continually surprised by the reference to "voluntary actions taken by companies" being brought up in discussion of the risks of AI, without some nuance given to why they would do that. The paragraph on surgical action goes in to about 5-10 times more detail on the potential issues with gov't regulation, implying to me that voluntary action is better. Even for someone at anthropic, i would hope that they would discuss it further.
I am genuinely curious to understand the incentives for companies who have the power to mitigate risk to actually do so. Are there good examples in the past of companies taking action that is harmful to their bottom line to mitigate societal risk of harm their products on society? My premise being that their primary motive is profit/growth, and that is revenue or investment dictated for mature and growth companies respectively (collectively "bottom line").
Im only in my mid 30s so dont have as much perspective on past examples of voluntary action of this sort with respect to tech or pre-tech corporates where there was concern of harm. Probably too late to this thread for replies, but ill think about it for the next time this comes up.
Major incentives currently in play are "PR fuckups are bad" and "if we don't curb our shit regulators will". Which often leads to things like "AI safety is when our AI doesn't generate porn when asked and refuses to say anything the media would be able to latch on to".
The rest is up to the companies themselves.
Anthropic seems to walk the talk, and has supported some AI regulation in the past. OpenAI and xAI don't want regulation to exist and aren't shy about it. OpenAI tunes very aggressively against PR risks, xAI barely cares, Google and Anthropic are much more balanced, although they lean towards heavy-handed and loose respectively.
China is its own basket case of "alignment is when what AI says is aligned to the party line", which is somehow even worse than the US side of things.
The framing of AI risk as a "rite of passage" resonates with me.
The "autonomy risks" section is what I think about most. We've seen our agents do unexpected things when given too much latitude. Not dangerous, just wrong in ways we didn't anticipate. The gap between "works in testing" and "works in production" is bigger than most people realize.
I'm less worried about the "power seizure" scenario than the economic disruption one. AI will take over more jobs as it gets better. There's no way around it. The question isn't whether, it's how we handle the transition and what people will do.
One thing I'd add: most engineers are still slow to adopt these tools. The constant "AI coding is bad" posts prove this while cutting-edge teams use it successfully every day. The adoption curve matters for how fast these risks actually materialize.
What makes you think that they will just keep improving? It's not obvious at all, we might soon hit a ceiling, if we haven not already - time will tell.
There are lots of technologies that have been 99% done for decades; it might be the same here.
From the essay - not presented in agreement (I'm still undecided), but Dario's opinion is probably the most relevant here:
> My co-founders at Anthropic and I were among the first to document and track the “scaling laws” of AI systems—the observation that as we add more compute and training tasks, AI systems get predictably better at essentially every cognitive skill we are able to measure. Every few months, public sentiment either becomes convinced that AI is “hitting a wall” or becomes excited about some new breakthrough that will “fundamentally change the game,” but the truth is that behind the volatility and public speculation, there has been a smooth, unyielding increase in AI’s cognitive capabilities.
> We are now at the point where AI models are beginning to make progress in solving unsolved mathematical problems, and are good enough at coding that some of the strongest engineers I’ve ever met are now handing over almost all their coding to AI. Three years ago, AI struggled with elementary school arithmetic problems and was barely capable of writing a single line of code. Similar rates of improvement are occurring across biological science, finance, physics, and a variety of agentic tasks. If the exponential continues—which is not certain, but now has a decade-long track record supporting it—then it cannot possibly be more than a few years before AI is better than humans at essentially everything.
> In fact, that picture probably underestimates the likely rate of progress. Because AI is now writing much of the code at Anthropic, it is already substantially accelerating the rate of our progress in building the next generation of AI systems. This feedback loop is gathering steam month by month, and may be only 1–2 years away from a point where the current generation of AI autonomously builds the next. This loop has already started, and will accelerate rapidly in the coming months and years. Watching the last 5 years of progress from within Anthropic, and looking at how even the next few months of models are shaping up, I can feel the pace of progress, and the clock ticking down.
What convinces me is this: I live in SF and have friends at various top labs, and even ignoring architecture improvements the common theme is this: any time researchers have spent time to improve understanding on some specific part of a domain (whether via SFT or RL or whatever), its always worked. Not superhuman, but measurable, repeatable improvements. In the words of sutskever, "these models.. they just wanna learn".
Inb4 all natural trends are sigmoidal or whatever, but so far, the trend is roughly linear, and we havent seen seen a trace of a plateau.
Theres the common argument that "Ghipiti 3 vs 4 was a much bigger step change" but its not if you consider the progression from much before, i.e. BERT and such, then it looks fairly linear /w a side of noise (fries).
Even if the technology doesn't get better, just imagine a world where all our processes are documented in a way that a computer can repeat them. And modifying the process requires nothing more than plain English or language.
What used to require specialized integration can now be accomplished by a generalized agent.
There is too much hand waving with respect to AI and their possible interactions in the physical world. Dario is definitely guilty of this. We currently discuss the economics of datacenters and gpu production, understanding very clearly the supply chain constraints, the bottlenecks, and the huge capital expenses representing them. On the other hand, we have entirely separate dialogues about AI risks which pretend none of these constraints exist. AI risk in the networked digital realm is a serious concern. However, I don't believe coordinated datacenters filled with autonomous AI pose a near-term physical expansionist threat. While they may be able to further optimize our supply chains, and usher in a similarly exponential growth in robotics -- people would have to hand hold and help instrument that physicality. I strongly believe such growth would be separate and significantly delayed from the LLM based intelligence gains we are currently experiencing and is likely decades away.
Geez, how is this comment so far down the list? Reading Dario's list of all the bad things AI could do, I kept asking myself "who would be so stupid as to give AI control of said instruments of destruction?" Dario writes as though the AI just assumes control of the physical world because it is SO POWERFUL.
I fear that when this technology grows up it will first be in the hands of the propagandists and war mongers. The rest of use won't stand a chance against the real-time propaganda streams convincing us why "we" needs to attack the bad guy country of the month die so we can take their stuff.
Or maybe we'll be so sedated by genAI, 24/7, always new, personally customized entertainment that we won't care.
Readit News
The AI in the book is almost feels like it is the main message masquerading as a subplot.
Asimov knew the risks, and I had assumed until fairly recently that the lessons and explorations that he had imparted into the Robot books had provided a level of cultural knowledge of what we were about to face. Perhaps the movie of I Robot was a warning of how much the signal had decayed.
I worry that we are sociologically unprepared, and sometimes it seems wilfully so.
People discussed this potential in great detail decades ago, Indeed the Sagan reference at the start of this post points to one of the significant contributors to the conversation, but it seems by the time it started happening, everyone had forgotten.
People are talking in terms of who to blame, what will be taken from me, and inevitability.
Any talk of a future we might want dismissed as idealistic or hype. Any depiction of a utopian future is met with derision far too often. Even worse the depiction can be warped to an evil caricature of "What they really meant".
How do we know what course to take if we can't talk about where we want to end up?
So what can you and I do? I know in my gut that imagining an ideal outcome won't change what actually happens, and neither will criticizing it really.
Shifts of dominant ideas can only come about through discussions. And sure, individuals can't control what happens. That's unrealistic in a world of billions. But each of us is invariably putting a little but of pressure in some direction. Ironically, you are doing that with your comment even while expressing the supposed futility of it. And overall, all these little pressures do add up.
Deleted Comment
Engage respectfully, Try and see other points of view, Try and express your point of view. I decided some time ago that I would attempt to continue conversations on here to try and at least get people to understand that other points of view could be held by rational people. It has certainly cost me Karma, but I hope there has been a small amount of influence. Quite often people do not change their minds by losing arguments, but by seeing other points of view and then given time to reflect.
>I know in my gut that imagining an ideal outcome won't change what actually happens
You might find that saying what you would like to see doesn't get heard, but you just have to remember that you can get anything you want at Alice's Restaurant (if that is not too oblique of a reference)
Talk about what you would like to see, If others would like to see that too, then they might join you.
I think most people working in AI are doing so in good faith and are doing what they think is best. There are plenty of voices telling them how not to it, many of those voices are contradictory. The instances of people saying what to do instead are much fewer.
If you declare that events are inevitable then you have lost. If you characterise Sam Altman as a sociopath playing the long game of hiding in research for years just waiting to pounce on the AI technology that nobody thought was imminent, then you have created a world in you mind where you cannot win. By imagining an adversary without morality it's easy to abdicate the responsibility of changing their mind, you can simply declare it can't be done. Once again choosing inevitability.
Perhaps try and imagine the world you want and just try and push a tiny fraction towards that world. If you are stuck in a seaside cave and the ocean is coming in, instead of pushing the ocean back, look to see if there is an exit at the other end, maybe there isn't one, but at least go looking for it, because if there is, that's how you find it.
Don't expect anyone building these systems to know what Bladerunner is, or "I have no mouth and I must scream" or any other great literature about the exact thing they are working on!
He never imagined, I suppose, that we would have the computing power necessary to just YOLO-dump the sum of all human knowledge into a few math problems and get really smart sounding responses generated in return.
The risks can be generalized well enough. Man’s hubris is its downfall etc etc.
But the specific issues we are dealing with have little to do with us feeling safe and protected behind some immutable rules that are built into the system.
>But the specific issues we are dealing with have little to do with us feeling safe and protected behind some immutable rules that are built into the system
If your interpretation of the Robot books was that was suggesting a few immutable rules would make us safe and protected, you may have missed the primary message. The overarching theme was an exploration of what those laws could do, and how they may not necessarily correlate with what we want or even perceive as safe and protected. If anything the rules represented a starting point and the books were presenting a challenge to come up with something better.
Anthropic's work on autoencoding activations down to measurable semantic points might prove a step towards that something better. The fact that they can do manipulations based upon those semantic points does suggest something akin to the laws of robotics might be possible.
When it comes to alignment, the way many describe it, it is simply impossible because humans themselves are not aligned. Picking a median, mean, or lowest common denominator of human alignment would be a choice that people probably cannot agree. We are unaligned on even how we could compromise.
In reality, if you have control over what AI does there are only two options.
1. We can make AI do what some people say,
2. We can make them do what they want (assuming we can make them want)
If we make them do what some people, that hands the power to those who have that say.
I think there will come a time when an AI will perceive people doing something wrong, that most people do not think is wrong, and the AI will be the one that is right. Do we want it to intervene or not? Are we instead happy with a nation developing superintelligence that is subservient to the wishes of say, Vladimir Putin.
He was idealistic even at the time. The 3 Laws were written 30 years after some of the earliest robots were aiming artillery barrages at human beings.
We aren't working 4 hour days because we no longer have to spend half the day waiting on things that were slower pre-internet. We're just supposed to deliver more, and oh, work more hours too since now you've always got your work with you.
Any discussion of today's AI firms has to start from the position of these companies being controlled by people deeply rooted in, and invested in, those systems and the negative application of that technology towards "working for a living" to date.
How do we get from there to a utopia?
"U.S workers just took home their smallest share of capital since 1947"
https://fortune.com/2026/01/13/us-workers-smallest-labor-sha...
How to get there:
1. Define the utopia in more detail.
2. Make the case that this is a preferable state. Make people want it.
3. Make the case that it is sustainable once achieved.
4. Identify specific differences between the preferred destination and where we are now.
5. Avoiding short term and temporary effects, work towards changing the differences to what the destination has. Even if that is only proclaiming that these changes are what you want
6. Show how those changes make us closer to the destination that people want.
Some of these are hard problems, I don't think any are intractable. I think they don't get done because they are hard, and opposing something is easier. Rather than building something you want, you can knock down something you don't like. Sure, that might get you closer to your desired state if you consider nothingness to be better than undesired, but without building you will never get there.
If you want everyone to live in a castle, build a castle and invite everybody over. If you start by destroying huts you will just be making adversaries. The converse is true also, if you want everyone to live in huts, build more huts and invite everyone over. If they don't come it's because you haven't made the case that it is a preferable state. Knocking down the castle is not going to convince them of that.
Forgetting that if you really can hear a dogwhistle, you're also a dog.
Dario's essay carefully avoids its own conclusion. He argues that AI will democratize mass casualty weapons (biology especially), that human coordination at civilizational scale is impossible, and that human-run surveillance states inevitably corrupt. But he stops short of the obvious synthesis: the only survivable path is an AI-administered panopticon.
That sounds dystopian until you think it through:
This is the Helios ending from Deus Ex, and it's the Culture series' premise. Benevolent AI sovereignty isn't necessarily dystopia, and it might be the only path to something like Star Trek.The reason we can't talk about this is that it's unspeakable from inside the system. Dario can't say it (he's an AI company CEO.) Politicians can't say it because it sounds insanely radical. So the discourse stays stuck on half-measures that everyone knows won't work.
I honestly believe this might be the future to work toward, because the alternatives are basically hell.
The current AI promise for them goes something like: "Oops this chittering machine will soon be able to do all you're good at and derive meaning from. But hey, at least you will end up homeless and part of a permanent underclass."
And the people building it are (rightfully) worried about it killing humanity. So why do we have to continue on this course again? An advanced society would at this point decide to pause what they are doing and reconsider.
I would like to believe that we're about to see a rapid proliferation of useful robots, but progress has been much slower with the physical world than with information-based tasks.
After the DARPA Urban Challenge of 2007, I thought that massive job losses from robotic car and truck drivers were only 5-8 years away. But in 2026 in the US only Waymo has highly autonomous driving systems, in only a few markets. Most embodied tasks don't even have that modest level of demonstrated capability.
I actually worry that legislators -- people with white collar jobs -- will overestimate the near-term capabilities of AI to handle jobs in general, and prematurely build solutions for a "world without work" that will be slow to arrive. (Like starting UBI too early instead of boosting job retraining, leaving health care systems understaffed for hands-on work.)
10 years ago I predicted that the uptake of autonomous vehicles would be slow but that it would be because of labor protections. While those have had some impact, that isn't really the issue: it's that the cars just don't quite work well enough yet and that last ~20% of function turns out to be both incredibly difficult and incredibly important.
But that feels like the least of the worries to me. There seems to be an implicit assumption that those physical lines of work don't get eroded by the higher proportion of able bodied people who are suddenly unemployable. Yes there is some training required etc. but the barriers to entry aren't so high that in the shortish to medium term you don’t see more people gravitating to those industries and competing wages further down to not make then sustainable employment long term. I'd even think that having LLMs that can recognise photos or understand fuzzily explain questions about some blue collar skills many have forgotten actually reduces the barrier even more
Even in the software world, the effect of being able to build software a lot faster isn't really leading to a fundamentally different software landscape. Yes, you can now pump out a month's worth of CRUD in a couple days, but ultimately it's just the same CRUD, and there's no reason to expect that this will change because of LLMs.
Of course, creative people with innovative ideas will be able to achieve more, a talented engineer will be able to embark on a project that they didn't have the time to build before, and that will likely lead to some kind of software surplus that the economy feels on the margins, but in practical terms the economy will continue to chug along at a sustained pace that's mostly inline with e.g. economic projections from 10 years ago.
Even just a year ago, most people thought the practical effects in software engineering were incremental too. It took another generation of models and tooling to get to the point where it could start having a large impact.
What makes you think the same will not happen in other knowledge-based fields after another iteration or two?
Hum... Are you saying it's having clear positive (never mind "transformative") impact somewhere? Can you point any place we can see observable clear positive impact?
The unfortunate truth (for Amodei) is you cant automate true creativity and nor standardise taste. Try as they might.
Deleted Comment
You clearly didn't read the post. He is talking about AI that is smarter than any human, not today's LLMs. The fact that powerful AI doesn't exist yet doesn't mean there is nothing to worry about.
This kind of petty remark is like a reverse em dash. Greetings fellow human.
Anyway, I did read it. The author's description of a future AI is basically just a more advanced version of LLMs
> By “powerful AI,” I have in mind an AI model—likely similar to today’s LLMs in form, though it might be based on a different architecture, might involve several interacting models, and might be trained differently—with the following properties:
They then go on to list several properties that meet their definition, but what I'm trying to explain in my comment is that I don't accept them all at face value. I think it's fair to critique from that perspective since the author explicitly modeled their future based on today's LLMs, unlike many AI essays that skip straight to the super intelligence meme as their premise.
- Prediction of exponential AI research feedback loops (AI coding speeding up AI R&D) which Amodei says is already starting today
- AI being a race between democracies and autocracies with winner-takes-all dynamics, with compute being crucial in this race and global slowdown being infeasible
- Mention of bioweapons and mirror life in particular being a big concern
- The belief that AI takeoff would be fast and broad enough to cause irreplaceable job losses rather than being a repeat of past disruptions (although this essay seems to be markedly more pessimistic than AI 2027 with regard to inequality after said job losses)
- Powerful AI in next few years, perhaps as early as 2027
I wonder if either work influenced the other in any way or is this just a case of "great minds think alike"?
Early "rationalist" community was concerned with AI in this way 20 years ago. Eliezer inspired and introduced the founders of Google DeepMind to Peter Thiel to get their funding. Altman acknowledged how influential Eliezer was by saying how he is most deserving of a Nobel Peace prize when AGI goes well (by lesswrong / "rationalist" discussion prompting OpenAI). Anthropic was a more X-risk concerned fork of OpenAI. Paul Christiano inventor of RLHF was big lesswrong member. AI 2027 is an ex-OpenAI lesswrong contributor and Scott Alexander, a centerpiece of lesswrong / "rationalism". Dario, Anthropic CEO, sister is married to Holden Karnofsky, a centerpiece of effective altruism, itself a branch of lesswrong / "rationalism". The origin of all this was directionally correct, but there was enough power, $, and "it's inevitable" to temporarily blind smart people for long enough.
But clearly if out of context someone said something like this:
"Clearly, the most obvious effect will be to greatly increase economic growth. The pace of advances in scientific research, biomedical innovation, manufacturing, supply chains, the efficiency of the financial system, and much more are almost guaranteed to lead to a much faster rate of economic growth. In Machines of Loving Grace, I suggest that a 10–20% sustained annual GDP growth rate may be possible."
I'd say that they were a snake oil salesman. All of my life experience says that there's no good reason to believe Dario's predictions here, but I'm taken in just as much as everyone else.
We're gonna die, but it's not going to be AI that does it: it'll be the oceans boiling and C3 carbon fixation flatlining that does it.
What is XRisk? I would have inductively thought adult but that doesn't sound right.
It used to be a small group of people who mostly just believed that AI is a very important technology overlooked by most. Now, they're vindicated, the importance of AI is widely understood, and the headcount in the industry is up x100. But those people who were on the ground floor are still there, they all know each other, and many keep in touch. And many who entered the field during the boom were those already on the periphery of the same core group.
Which is how you get various researchers and executives who don't see eye to eye anymore but still agree on many of the fundamentals - or even things that appear to an outsider as extreme views. They may have agreed on them back in year 2010.
"AGI is possible, powerful, dangerous" is a fringe view in the public opinion - but in the AI scene, it's the mainstream view. They argue the specifics, not the premise.
I am genuinely curious to understand the incentives for companies who have the power to mitigate risk to actually do so. Are there good examples in the past of companies taking action that is harmful to their bottom line to mitigate societal risk of harm their products on society? My premise being that their primary motive is profit/growth, and that is revenue or investment dictated for mature and growth companies respectively (collectively "bottom line").
Im only in my mid 30s so dont have as much perspective on past examples of voluntary action of this sort with respect to tech or pre-tech corporates where there was concern of harm. Probably too late to this thread for replies, but ill think about it for the next time this comes up.
The rest is up to the companies themselves.
Anthropic seems to walk the talk, and has supported some AI regulation in the past. OpenAI and xAI don't want regulation to exist and aren't shy about it. OpenAI tunes very aggressively against PR risks, xAI barely cares, Google and Anthropic are much more balanced, although they lean towards heavy-handed and loose respectively.
China is its own basket case of "alignment is when what AI says is aligned to the party line", which is somehow even worse than the US side of things.
The "autonomy risks" section is what I think about most. We've seen our agents do unexpected things when given too much latitude. Not dangerous, just wrong in ways we didn't anticipate. The gap between "works in testing" and "works in production" is bigger than most people realize.
I'm less worried about the "power seizure" scenario than the economic disruption one. AI will take over more jobs as it gets better. There's no way around it. The question isn't whether, it's how we handle the transition and what people will do.
One thing I'd add: most engineers are still slow to adopt these tools. The constant "AI coding is bad" posts prove this while cutting-edge teams use it successfully every day. The adoption curve matters for how fast these risks actually materialize.
There are lots of technologies that have been 99% done for decades; it might be the same here.
> My co-founders at Anthropic and I were among the first to document and track the “scaling laws” of AI systems—the observation that as we add more compute and training tasks, AI systems get predictably better at essentially every cognitive skill we are able to measure. Every few months, public sentiment either becomes convinced that AI is “hitting a wall” or becomes excited about some new breakthrough that will “fundamentally change the game,” but the truth is that behind the volatility and public speculation, there has been a smooth, unyielding increase in AI’s cognitive capabilities.
> We are now at the point where AI models are beginning to make progress in solving unsolved mathematical problems, and are good enough at coding that some of the strongest engineers I’ve ever met are now handing over almost all their coding to AI. Three years ago, AI struggled with elementary school arithmetic problems and was barely capable of writing a single line of code. Similar rates of improvement are occurring across biological science, finance, physics, and a variety of agentic tasks. If the exponential continues—which is not certain, but now has a decade-long track record supporting it—then it cannot possibly be more than a few years before AI is better than humans at essentially everything.
> In fact, that picture probably underestimates the likely rate of progress. Because AI is now writing much of the code at Anthropic, it is already substantially accelerating the rate of our progress in building the next generation of AI systems. This feedback loop is gathering steam month by month, and may be only 1–2 years away from a point where the current generation of AI autonomously builds the next. This loop has already started, and will accelerate rapidly in the coming months and years. Watching the last 5 years of progress from within Anthropic, and looking at how even the next few months of models are shaping up, I can feel the pace of progress, and the clock ticking down.
What convinces me is this: I live in SF and have friends at various top labs, and even ignoring architecture improvements the common theme is this: any time researchers have spent time to improve understanding on some specific part of a domain (whether via SFT or RL or whatever), its always worked. Not superhuman, but measurable, repeatable improvements. In the words of sutskever, "these models.. they just wanna learn".
Inb4 all natural trends are sigmoidal or whatever, but so far, the trend is roughly linear, and we havent seen seen a trace of a plateau.
Theres the common argument that "Ghipiti 3 vs 4 was a much bigger step change" but its not if you consider the progression from much before, i.e. BERT and such, then it looks fairly linear /w a side of noise (fries).
Bicycles? carbon fiber frames, electronic shifting, tubeless tires, disc brakes, aerodynamic research
Screwdrivers? impact drivers, torque-limiting mechanisms, ergonomic handles
Glass? gorilla glass, smart glass, low-e coatings
Tires? run-flats, self-sealing, noise reduction
Hell even social technologies improve!
How is a technology "done?"
What used to require specialized integration can now be accomplished by a generalized agent.