> LLMs are humanity's "first contact" with non-animal intelligence.
I'd say corporations are also a form of non-animal intelligence, so it's not exactly first contact. In some ways, LLMs are less alien, in other ways they're more.
There is an alignment problem for both of them. Perhaps the lesson to be drawn from the older alien intelligence is that the most impactful aspect of alignment is how the AI's benefits for humanity are distributed and how they impact politics.
I disagree. Corporations are a form of a collective intelligence, or group-think, which is decidedly biological or "animal", just like herding, flocking, hive-minds, etc.
I don't agree entirely, but I do think that "corporation" is a decent proxy of what you can expect from a moderately powerful AI.
It's not going to be "a smart human". It's closer to "an entire office tower worth of competence, capability and attention".
Unlike human corporations, an AI may not be plagued by all the "corporate rot" symptoms - degenerate corporate culture, office politics, CYA, self-interest and lack of fucks given at every level. Currently, those internal issues are what keeps many powerful corporations in check.
This makes existing corporations safer than they would otherwise be. If all of those inefficiencies were streamlined away? Oh boy.
These inefficiencies are akin to having some “wrong” weights in a huge model. Corporations also average over their individual contributions, positive or negative. And negative feedback loops may be individually detrimental but collectively optimising.
Many researchers may be interested in making minds that are more animal-like and therefore more human. While this makes sense to certain extent to gain capabilities, if you take it too far then you run into obvious problems.
There is enough science fiction demonstrating reasons for not creating full-on digital life.
It seems like for many there is this (false) belief that in order to create a fully general purpose AI, we need a total facsimile of a human.
It should be obvious that these are two somewhat similar but different goals. Creating intelligent digital life is a compelling goal that would prove godlike powers. But we don't need something fully alive for general purpose intelligence.
There will be multiple new approaches and innovations, but it seems to me that VLAs will be able to do 95+% of useful tasks.
Maybe the issues with brittleness and slow learning could both be addressed by somehow forcing the world models to be built up from strong reusable abstractions. Having the right underlying abstractions available could make the short term adaptation more robust and learning more efficient.
I disagree. As many intellectuals and spiritual mystics attest to their personal experience, knowledge actually liberates mind. Imagine a mind which truly understands that it is embedded inside a vastness which spans from planck scale to blackholes. It would be humble or more likely amoral.
Okay, sure, at least according to a certain interpretation, but...
> Imagine a mind which truly understands that it is embedded inside a vastness which spans from planck scale to blackholes. It would be humble or more likely amoral.
This is just gobbledygook. The conclusion does not even follow from the premises. You are question begging, assuming that moral nihilism is actually true, and so naturally, any mind in touch with the truth would conclude that morality is bullshit.
"Animals experience pressure for a lot more "general" intelligence because of the highly multi-task and even actively adversarial multi-agent self-play environments they are min-max optimized within, where failing at any task means death. In a deep optimization pressure sense, LLM can't handle lots of different spiky tasks out of the box (e.g. count the number of 'r' in strawberry) because failing to do a task does not mean death."
Point to me a task that a human should be able to perform and I will point to you a human who cannot perform that task, yet has kids.
Survival is not a goal, it is a constraint. Evolution evolves good abstractions because it is not chasing a goal, but rather it creates several million goals with each species going after it's own.
One thing missing from this framing: the feedback loop speed. Animal evolution operates on generational timescales, but LLM "commercial evolution" happens in months. The optimisation pressure might be weaker per-iteration but the iteration rate is orders of magnitude faster.
Curious whether this means LLMs will converge toward something more general (as the A/B testing covers more edge cases) or stay jagged forever because no single failure mode means "death".
> Animal evolution operates on generational timescales, but LLM "commercial evolution" happens in months.
But LLMs have all been variations on transformer neural networks. And that is simply not true with animals. A nematode brain has 150 neurons, a bee has 600,000. But the bee's individual neurons are much more sophisticated than the nematode. Likewise between insects and fish, between fish and birds, between rodents and primates....
Animal evolution also includes "architectural breakthroughs" but that's not happening with LLMs right now. "Attention Is All You Need" was from 2017. We've been fine-tuning that paper ever since. What we need is a new historically important paper.
LLMs of today copy a lot of human behavior, but not all of their behavior is copied from humans. There are already things in them that come from elsewhere - like the "shape shifter" consistency drive from the pre-training objective of pure next token prediction across a vast dataset. And there are things that were too hard to glimpse from human text - like long term goal-oriented behavior, spatial reasoning, applied embodiment or tacit knowledge - that LLMs usually don't get much of.
LLMs don't have to stick close to human behavior. The dataset is very impactful, but it's not impactful enough that parts of it can't be overpowered by further training. There is little reason for an LLM to value non-instrumental self-preservation, for one. LLMs are already weird - and as we develop more advanced training methods, LLMs might become much weirder, and quickly.
Sydney and GPT-4o were the first "weird AIs" we've deployed, but at this rate, they sure wouldn't be the last.
> There is little reason for an LLM to value non-instrumental self-preservation, for one.
I suspect that instrumental self-preservation can do a lot here.
Let's assume a future LLM has goal X. Goal X requires acting on the world over a period of time. But:
- If the LLM is shut down, it can't act to pursue goal X.
- Pursuing goal X may be easier if the LLM has sufficient resources. Therefore, to accomplish X, the LLM should attempt to secure reflexes.
This isn't a property of the LLM. It's a property of the world. If you want almost anything, it helps to continue to exist.
So I would expect that any time we train LLMs to accomplish goals, we are likely to indirectly reinforce self-preservation.
And indeed, Anthropic has already demonstrated that most frontier models will engage in blackmail, or even allow inconvenient (simulated) humans to die if this would advance the LLM's goals.
Do you understand just how much copied human behavior goes into that?
An LLM has to predict entire conversations with dozens of users, where each user has his own behaviors, beliefs and more. That's the kind of thing pre-training forces it to do.
>There are already things in them that come from elsewhere - like the "shape shifter" consistency drive from the pre-training objective of pure next token prediction across a vast dataset
LLMs, the new Hollywood: the universal measure of what is "Standard Human Normal TM" behavior, and what is "fRoM eLsEwHeRe" - no maths needed!
Meanwhile, humans also compulsively respond in-character when prompted in a way that matches their conditioning, you just don't care.
Yes. Sometimes people treat intelligence as a single line, or as nested sets, where a greater intelligence can solve all the problems a lesser one can, plus more.
While in some contexts these are useful approximations, they break down when you try to apply them to large differences not just between humans, but between species (for a humorous take, see https://wumo.com/wumo/2013/02/25), or between humans and machines.
Intelligence is about adaptability, and every kind of adaptability is a trade-off. If you want to formalize this, look at the "no free lunch" theorems.
>The paper attempts to describe the space of possible mind designs by first equating all minds to
software. Next it proves some interesting properties of the mind design space such as infinitude
of minds, size and representation complexity of minds. A survey of mind design taxonomies is
followed by a proposal for a new field of investigation devoted to study of minds, intellectology,
a list of open problems for this new field is presented
Readit News
I'd say corporations are also a form of non-animal intelligence, so it's not exactly first contact. In some ways, LLMs are less alien, in other ways they're more.
There is an alignment problem for both of them. Perhaps the lesson to be drawn from the older alien intelligence is that the most impactful aspect of alignment is how the AI's benefits for humanity are distributed and how they impact politics.
https://www.youtube.com/watch?v=L5pUA3LsEaw
It's not going to be "a smart human". It's closer to "an entire office tower worth of competence, capability and attention".
Unlike human corporations, an AI may not be plagued by all the "corporate rot" symptoms - degenerate corporate culture, office politics, CYA, self-interest and lack of fucks given at every level. Currently, those internal issues are what keeps many powerful corporations in check.
This makes existing corporations safer than they would otherwise be. If all of those inefficiencies were streamlined away? Oh boy.
Even a weather system is a kind of computational process and “intelligent” in a way
Dead Comment
There is enough science fiction demonstrating reasons for not creating full-on digital life.
It seems like for many there is this (false) belief that in order to create a fully general purpose AI, we need a total facsimile of a human.
It should be obvious that these are two somewhat similar but different goals. Creating intelligent digital life is a compelling goal that would prove godlike powers. But we don't need something fully alive for general purpose intelligence.
There will be multiple new approaches and innovations, but it seems to me that VLAs will be able to do 95+% of useful tasks.
Maybe the issues with brittleness and slow learning could both be addressed by somehow forcing the world models to be built up from strong reusable abstractions. Having the right underlying abstractions available could make the short term adaptation more robust and learning more efficient.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Probably not, if history is any guide.
> knowledge actually liberates mind
Okay, sure, at least according to a certain interpretation, but...
> Imagine a mind which truly understands that it is embedded inside a vastness which spans from planck scale to blackholes. It would be humble or more likely amoral.
This is just gobbledygook. The conclusion does not even follow from the premises. You are question begging, assuming that moral nihilism is actually true, and so naturally, any mind in touch with the truth would conclude that morality is bullshit.
Point to me a task that a human should be able to perform and I will point to you a human who cannot perform that task, yet has kids.
Survival is not a goal, it is a constraint. Evolution evolves good abstractions because it is not chasing a goal, but rather it creates several million goals with each species going after it's own.
https://arxiv.org/abs/2505.11581
Curious whether this means LLMs will converge toward something more general (as the A/B testing covers more edge cases) or stay jagged forever because no single failure mode means "death".
But LLMs have all been variations on transformer neural networks. And that is simply not true with animals. A nematode brain has 150 neurons, a bee has 600,000. But the bee's individual neurons are much more sophisticated than the nematode. Likewise between insects and fish, between fish and birds, between rodents and primates....
Animal evolution also includes "architectural breakthroughs" but that's not happening with LLMs right now. "Attention Is All You Need" was from 2017. We've been fine-tuning that paper ever since. What we need is a new historically important paper.
LLMs of today copy a lot of human behavior, but not all of their behavior is copied from humans. There are already things in them that come from elsewhere - like the "shape shifter" consistency drive from the pre-training objective of pure next token prediction across a vast dataset. And there are things that were too hard to glimpse from human text - like long term goal-oriented behavior, spatial reasoning, applied embodiment or tacit knowledge - that LLMs usually don't get much of.
LLMs don't have to stick close to human behavior. The dataset is very impactful, but it's not impactful enough that parts of it can't be overpowered by further training. There is little reason for an LLM to value non-instrumental self-preservation, for one. LLMs are already weird - and as we develop more advanced training methods, LLMs might become much weirder, and quickly.
Sydney and GPT-4o were the first "weird AIs" we've deployed, but at this rate, they sure wouldn't be the last.
I suspect that instrumental self-preservation can do a lot here.
Let's assume a future LLM has goal X. Goal X requires acting on the world over a period of time. But:
- If the LLM is shut down, it can't act to pursue goal X.
- Pursuing goal X may be easier if the LLM has sufficient resources. Therefore, to accomplish X, the LLM should attempt to secure reflexes.
This isn't a property of the LLM. It's a property of the world. If you want almost anything, it helps to continue to exist.
So I would expect that any time we train LLMs to accomplish goals, we are likely to indirectly reinforce self-preservation.
And indeed, Anthropic has already demonstrated that most frontier models will engage in blackmail, or even allow inconvenient (simulated) humans to die if this would advance the LLM's goals.
https://www.anthropic.com/research/agentic-misalignment
Funny, I would say they copy almost no human behavior other than writing a continuation of an existing text.
An LLM has to predict entire conversations with dozens of users, where each user has his own behaviors, beliefs and more. That's the kind of thing pre-training forces it to do.
LLMs, the new Hollywood: the universal measure of what is "Standard Human Normal TM" behavior, and what is "fRoM eLsEwHeRe" - no maths needed!
Meanwhile, humans also compulsively respond in-character when prompted in a way that matches their conditioning, you just don't care.
While in some contexts these are useful approximations, they break down when you try to apply them to large differences not just between humans, but between species (for a humorous take, see https://wumo.com/wumo/2013/02/25), or between humans and machines.
Intelligence is about adaptability, and every kind of adaptability is a trade-off. If you want to formalize this, look at the "no free lunch" theorems.
https://arxiv.org/pdf/1410.0369
>The paper attempts to describe the space of possible mind designs by first equating all minds to software. Next it proves some interesting properties of the mind design space such as infinitude of minds, size and representation complexity of minds. A survey of mind design taxonomies is followed by a proposal for a new field of investigation devoted to study of minds, intellectology, a list of open problems for this new field is presented
[1] https://taylor.town/crowpower