I agree with Hinton, although a lot hinges on your definition of "understand."
I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go.
Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they found is that there isn't really a limit to how good it can get. There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train. And it can go far beyond what the best human Go player can ever do. Even though very smart humans have spent their lifetimes deeply studying the game, and Alpha Zero had to learn everything from scratch.
One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move. And that model is obviously very sophisticated, and if you are going to say that it doesn't understand the game of Go, I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.
Likewise, with large language models, while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead. As Hinton says, there is no real limit to how sophisticated they can get. When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains. And the closer it gets, the more sophisticated model it needs. In the sense that Alpha Zero needs to "understand" the game of Go to play effectively, the large language model needs to understand "the world" to get better at predicting.
The difference is that "the world" is not exhaustible in the same way as Go is. While it's surely true that the number of possible overall Go game states is extremely large, the game itself is trivially representable as a set of legal moves and rules. The "world model" of the Go board is actually just already exhaustive and finite, and the computer's work in playing against itself is to generate more varied data within that model rather than to develop that model itself. We know that when Alpha Zero plays a game against itself it is valuable data because it is a legitimate game which most likely represents a new situation it hasn't seen before and thus expands its capacity.
For an LLM, this is not even close to being the case. The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. And we certainly haven't figured out a "model" which would let a computer generate new and valid data that expands its understanding of the world beyond its inputs, so self-training is a non-starter for LLMs. What the LLM is "understanding", and what it is reinforced to "understand" is not the world but the format of texts, and while it may get very good at understanding the format of texts, that isn't equivalent to an understanding of the world.
>The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility.
No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts.
To the bird who can intuitively sense and use electromagnetic waves for motion and guidance, your model of the world is fundamentally incomplete.
There is a projection of the world in text. Moreover training on additional modalities is trivial for a transformer. That's all that matters.
Kant would like a word with you about your point on whether people themselves understand the world and not just the format of their perceptions... :)
I think if you're going to be strict about this, you have to defend against the point of view that the same 'ding an sich' problem applies to both LLMs and people. And also whether if you had a limit sequence of KL divergences, one from a person's POV of the world, and one from an LLM's POV of texts, what it is about how a person approaches better grasp of reality - and likewise their KL divergence approaches 0, in some sense implying that their world model is becoming the same as the distribution of the world - that can only apply to people.
It seems possible to me that there is probably a great deal of lurking anthropocentrism that humanity is going to start noticing more and more in ourselves in the coming years, probably in both the direction of AI and the direction of other animals as we start to understand both better
The world on our plane of existence absolutely is exhaustible, just on a much, much larger scale. Doesn't mean that the process is fundamentally different, and for the human perspective there might be diminishing returns.
LLMs are very good at uncovering the mathematical relationships between words, many layers deep. Calling that understanding is a claim about what understanding is. But because we know how the LLMs we're talking about at the moment are trained, it seems to have more problems:
LLMs do not directly model the world; they train on and model what people write about the world. It is an AI model of a computed gestalt human model of the world, rather than a model of the world directly. If you ask it a question, it tells you what it models someone else (a gestalt of human writing) is most likely say. That in turn is strengthened if user interaction accepts it and corrected only if someone tells it something different.
If we were to define that as what "understanding" is, we would equivalently be saying that a human bullshit artist would have expert understanding if only they produced more believable bullshit. (They also just "try to sound like an expert".)
Likewise, I'm not convinced that we can measure its understanding just by identifying inaccuracies or measuring the difference between its answers and expert answers -
There would be no difference between bluffing your way through the interview (relying on your interviewer's limitations in how they interrogate you) and acing the interview.
There seems to be a fundamental difference in levels of indirection. Where we "map the territory", LLMs "map the maps of the territory".
It can be an arbitrarily good approximation, and practically very useful, but it's a strong ontological step to say one thing "is" another just because it can be used like it.
"LLMs do not directly model the world; they train on and model what people write about the world"
This is true. But human brains don't directly model the world either, they form an internal model based on what comes in through their senses. Humans have the advantage of being more "multi-modal," but that doesn't mean that they get more information or better information.
Much of my "modeling of the world" comes from the fact that I've read a lot of text. But of course I haven't read even a tiny fraction of what GPT4 has.
That said, LLMs can already train on images, as GPT4-V does. And the image generators as well do this, it's just a matter of time before the two are fully integrated. Later we'll see a lot more training on video and sound, and it all being integrated into a single model.
We could anthropomorphize any textbook too and claim it has human level understanding of the subject. We could then claim the second edition of the textbook understands the subject better than the first. Anyone who claims the LLM "understands" is doing exactly this. What makes the LLM more absurd though is the LLM will actually tell you it doesn't understand anything while a book remains silent but people want to pretend we are living in the Matrix and the LLM is alive.
Most arguments then descend into confusing the human knowledge embedded in a textbook with the human agency to apply the embedded knowledge. Software that extracts the knowledge from all textbooks has nothing to do with the human agency to use that knowledge.
I love chatGPT4 and had signed up in the first few hours it was released but I actually canceled my subscription yesterday. Part because of the bullshit with the company these past few days but also because it had just become a waste of time the past few months for me. I learned so much this year but I hit a wall that to make any progress I need to read the textbooks on the subjects I am interested in just like I had to this time last year before chatGPT.
We also shouldn't forget that children anthropomorphize toys and dolls quite naturally. It is entirely natural to anthropomorphize a LLM and especially when it is designed to pretend it is typing back a response like a human would. It is not bullshitting you though when it pretends to type back a response about how it doesn't actually understand what it is writing.
> One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move.
It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move (or moves that are better than others) and hence a function that goes from the state of the game to the best move. We just don't have a way to compute this function, but it exists. And that function doesn't need the concept of lookahead, that's just an intuitive way of how could find some of its values. Likewise ML algorithms don't necessarily need lookahead, they can just try to approximate that function with enough precision by exploiting patterns in it. And that's why we can still craft puzzles that some AIs can't solve but humans can, by exploiting edge cases in that function that the ML algorithm didn't notice but are solvable with understanding of the game.
The thing is though, does this really matter if eventually we won't be able to notice the difference?
> It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move
Is there really a difference between the two? If a certain move shapes the opponent's remaining possible moves into a smaller subset, hasn't AlphaGo "looked ahead"? In other words, when humans strategize and predict what happens in the real world, aren't they doing the same thing?
I suppose you could argue that humans also include additional world models in their planning, but it's not clear to me that these models are missing and impossible for machine learning models to generate during training.
> Since Go is a deterministic game there is always a best move
The rules of the game are deterministic, but you may be going a step too far with that claim.
Is the game deterministic when your opponent is non-deterministic?
Is there an optimal move for any board state given that various opponents have varying strategies? What may be the best move against one opponent may not be the best move against another opponent.
> to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move.
While I imagine alpha go does some brute force and some tree exploration, I think the main "intelligent" component of alpha go is the ability to recognize a "good" game state from a "bad" game state based on that moment in time, not any future plans or possibilities. That pattern recognition is all it has once its planning algorithm has reached the leaves of the trees. Correct me if I'm wrong, but I doubt alpha go has a neural net evaluating an entire tree of moves all at once to discover meta strategies like "the opponent focusing on this area" or "the opponent feeling on the back foot."
You can therefore imagine a pattern recognition algorithm so good that it is able to pick a move by only looking 1 move into the future, based solely on local stone densities and structures. Just play wherever improves the board state the most. It does not even need to "understand" that a game is being played.
> while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead.
So I don't think this statement is necessarily true. "Understanding" is a major achievement, but I don't think it requires planning. A computer can understand that 2+2=4 or where to play in tic-tac-toe without any "planning".
That said, there's probably not much special about the concept of planning either. If it's just simulating a tree of future possibilities and pruning it based on evaluation, then many algorithms have already achieved that.
The "meta" here is just the probability distribution of stone densities. The only way it can process those is by monte Carlo simulation. The DNN (trained by reinforcement learning) evaluates the simulations and outputs the top move(s).
> As Hinton says, there is no real limit to how sophisticated they can get.
There’s no limit to how sophisticated a model can get, but,
1. That’s a property shared with many architectures, and not really that interesting,
2. There are limits to the specific ways that we train models,
3. We care about the relative improvement that these models deliver, for a given investment of time and money.
From a mathematical perspective, you can just kind of keep multiplying the size of your model, and you can prove that it can represent arbitrary complicated structures (like, internal mental models of the world). That doesn’t mean that your training methods will produce those complicated structures.
With Go, I can see how the model itself can be used to generate new, useful training data. How such a technique could be applied to LLMs is less clear, and its benefits are more dubious.
A big difference between a game like Go and writing text is that text is single player. I can write out the entire text, look at it and see where I made mistakes on the whole and edit those. I can't go back in a game of Go and change one of my moves that turned out to be a mistake.
So trying to make an AI that solves the entire problem before writing the first letter will likely not result in a good solution while also making it compute way too much since it solves the entire problem for every token generated. That is the kind of AI we know how to train so for now that is what we have to live with, but it isn't the kind of AI that would be efficient or smart.
This doesn't seem like a major difference, since LLMs are also choosing from a probability distribution of tokens for the most likely one, which is why they respond a token at a time. They can't "write out' the entire text at a time, which is why fascinating methods like "think step by step" work at all.
> There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train
I don’t think that’s a given. AlphaZero may have found an extremely high local optimum that isn’t the global optimum.
When playing only against itself, it won’t be able to get out of that local optimum, and when getting closer and closer to it even may ‘forget’ how to play against players that make moves that AplhaGo never would make, and that may be sufficient for a human to beat it (something like that happened with computer chess in the early years, where players would figure out which board positions computers were bad at, and try to get such positions on the board)
I think you have to keep letting it play against other good players (human or computer) that play differently to have it keep improving, and even then, there’s no guarantee it will find a global optimum.
Alphazero runs monte carlo tree search so it has a next move "planning" simulator. This computes the probability that specific moves up to some distance lead to a win.
LLMs do not have a "planning" module or simulator. There is no way the LLM can plan.
Could build a planning system into an LLM? Possibly and probably, but that is still open research. LeCunn is trying to figure out how to train them effectively. But even an LLM with a planning system does not make it AGI.
Some will argue that iteratively feeding the output embedding back into the input will retain the context but even in those cases it rapidly diverges or as we say "hallucinates"... still happens even with large input context windows. So there is still no planning here and no world model or understanding.
The issue with Alpha Zero analogy extremes is that those are extremely constrained conditions, so can't be generalized to something infinitely more complicated like speech
And
> When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains.
For example, it can reach 25% of accuracy and have an math limit of 26%, so "forever getting closer to 100% with time" would still result in a waste of even infinite resources
Next word generation is one way to put it. The key point here is we have no idea what’s happening in the black box that is the neural network. It could be forming very strong connections between concepts in there with multi tiered abstractions.
> I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.
Is it just me or does this read like “here is my assumption about what you said, and now here is my passive aggressive judgement about that assumption”? If you’re not certain about what they mean by the word “understand”, I bet you could ask and they might explain it. Just a suggestion.
I've asked that question in the past and I've never gotten an answer. Some people sidestep the question by describing something or other that they're confident isn't understanding; others just decline to engage entirely, asserting that the idea is too ridiculous to take seriously. In my experience, people with a clear idea of what they mean by the word "understand" are comfortable saying that ML models understand things.
The fallacy being made in this argument is that computers need to perform tasks the same way as humans to achieve equal or better performance on them. While having better "system 2" abilities may improve performance, it's plausible that scaled-up next-token prediction along with a bit of scaffolding and finetuning could match human performance on the same diversity of tasks while doing them a completely different way.
If I had to critique Hinton's claims, I would say his usage of the word "understand" can be vague and communicate assumptions because it's from an ontology used for reasoning about human reasoning, not this new alien form of reasoning which language models embody.
I believe it was Feynman who said something to the effect of "airplanes do not fly like birds do, but they fly much faster and can carry much more". So yes, we do not need to exactly replicate how humans do things in order to do human-like things in a useful manner. Planes do not flap their wings, but the jet engine (which is completely unnatural) does a great job of making things fly when paired with fixed wings of a certain shape.
In addition to that, the "system 2" abilities might already be there with "epi" strategies like chain-of-thought prompting. Talking / writing to yourself might not be the most efficient way to think but at least I do it often enough when pondering a problem.
The argument “a sufficiently capable autocomplete must contain a level of general intelligence” is correct but also not very useful. It is a lot like saying “a sufficiently fast horse can fly”.
It is technically correct that when you take things to the extreme you can accomplish great things, but we may not reach those levels. We may require completely different technology to reach those levels of autocomplete, and we have simply reached a new plateau at this point in time.
The argument is simpler than that. Prediction requires a model, completely accurate or not. There's a projection of the world in text. A model of the text data we feed it is a model of the world as humans see it. The trend of loss is more and more accurate models of the dataset. So it won't stop at any arbitrary competency level. Indeed, there are already a few abilities GPT possess that are deemed Super Human. It's not a distinction that matters to the machine. It's all just data to be modelled.
We have reached those levels lol. That's why we're having this argument.
I think the trouble is that "model" is a very general term. If you had a computer doing simulations of artillery shots back in the 50s, then it would have a "model" of the world in terms of variables tracking projectiles, but this model doesn't generalize to anything else. If a computer does image recognition from the 90s and 2000s to recognize faces, then the computer has a "model" of visual information in the world, but this model only lets it recognize faces.
ChatGPT has a model of all the text information on the internet, but it remains to be seen what the hard limits of this model are. Does this model let it do logic or predict the future well, or will no amount of training give it those abilities? Simply being good in one task doesn't imply a general ability to do everything, or even most of everything. LLM's would simply be the last advancement in a field with a lot of similar advancements.
I've noticed that when I speak I really don't control each word.
I have an idea that I want to convey, but how each word comes to my mind as I form a sentence has always felt like it's controlled by an unconscious algorithm.
So I don't understand why people find this prediction mechanism so alien.
It isn't clear to me how much of communication is really in our control.
With the current tools, it feels like we still provide the ideas we want the AI to convey, and it may be using a nearly identical mechanism to us to form the words.
Consciousness would be the computer being able to come up with the ideas.
So, it seems to me we've gotten close enough on the communication side of intelligence.
But the machine is not conscious. When it is, it seems like it will generate its own ideas.
Are people debating whether the machine is conscious?
Otherwise, it feels very straightforward to grasp what we've made up to now.
Here is a question: What is the practical significance of viewing "AI" as autocomplete versus some other idea. Why try to influence how others view using a computer. Why anthromorphise. These are questions for which I have answers, but of course they are personal opinions. Historically, programmers often like to refer to programming as "magic". But magic is illusion, entertaintainment, tricks. Believing in "magic" is a personal choice.
Why not describe things in terms of what they do instead of what they "are". The latter is highly subjective and open to abuse.
NB. By "things" I mean software and the type of vacuous companies discussed on HN, not people (a bizarre comparison). For example, websites that go on and on about some so-called "tech" copmany but never once tell the reader what the company does. Or silly memes like "It's X for Y". What does it do and how does it work are questions that often go unasked and unanswered.
A few days ago someone related a story of working for a company that produced some software it claimed used "AI" but according to the commenter it used nothing more than regular expressions. Was ELIZA "AI". Maybe we should ask what isn't "AI". What happens with "magic" if the audience knows how the trick is performed.
> Why not describe things in terms of what they do instead of what they "are".
Would you say that about your spouse? The beauty of beholding one's wife is in who and what she is. What she does tells us something about who and what she is, to be sure, but any attempt to suppress the what (and the who) is dehumanizing and objectifying.
But, of course, what a thing does depends on what that thing is.
The reason I can say a human being can sort a list of numbers is because human beings have intention. When a human being sorts of list of numbers, they intend to sort the list. The intention is the cause and explanation for the actions taken the lead to a list of ordered numbers, as well as the resulting list of ordered numbers.
Does a computer sort numbers? In common speech, we say it does, just as we use all sorts of anthropomorphizing language when discussing computers. But at best, this is loose and analogical language. That's fine, as far as it goes, as long as we don't take it or mean it literally. However, the computer itself lacks intention. It is our intention that produces the computer, and our intention that makes the computer an instrument used by us to sort. Taken by itself, the computer is undergoing a transformation that effects something that we may interpreted as a list of sorted numbers, but the computer itself is not sorting. You wouldn't say that the clouds add x and y when x liters of water falls into a pool of y liters.
> The latter is highly subjective and open to abuse.
On the contrary, what a thing is is the most real and objective thing there is. An effect cannot be understood without knowing the cause, and the cause cannot be understood without knowing the agent. You can know some things about the effect, sure, and here the effect is that the text produced may be interpreted as intelligible. But the apparent intelligibility is borrowed from the source text, perhaps just a clever trick.
Current language models fail in all sorts of quantifiable ways, but I think that trying to discuss their merits away by reasoning about what it means to 'truly understand' something, or to be 'truly intelligent' is a complete dead-end.
It seems to me that it's based on the magical thought that there's something truly special and unique about us humans, as compared to other species or technology. Those discussions always seem more theological than science-driven to me. If you want to measure the difference between human performance and LLMs, there's a million experiments you can run. I'll gladly be convinced by data, and I'm open to the possibility that the data might point in either direction, or be inconclusive. But grand words without data are not convincing.
It's not magical; it's just agnostic. Some AI believers appear to be quite confident in their understanding of how the human brain works, despite the fact that those who have dedicated their entire lives to studying it will be the first to tell you that they ultimately have no idea.
I don’t see how this article even responds to the quote. Hinton didn’t make any claims that because it’s autocomplete it’s not thinking. If anything he’s saying really truly good autocomplete necessarily takes more understanding/thinking than a derogatory interpretation of ‘autocomplete’ would suggest.
Somehow OP seemed to twist that into “because I think on autopilot most of the time, then chatbots must think too”. Which is not totally incongruous with Hinton’s quote so much as a weird thing to balloon into an essay.
Yeah, it's a very silly article with wrong mathematical reasoning. Hinton is quite obviously talking about a much more information-theoretic approach to the process, but he's phrasing it in people-friendly terms.
What's a little more concerning to me is that people are reading and upvoting it. I think, because I have hopes and aspirations about working on some very hard problems and communicating them to the public at some point. And if this is the level of 'ooh, squirrel'! that we're going at, that the work that I make might get overshadowed by something really silly.
Perhaps an odd insecurity, but there it is, I think.
Yeah I agree with that for sure. It’s so strange how the majority of the research folks appear to be on this ‘new shiny’ mentality at the expense of fundamentals. Especially for how new this field is, relatively. It’s not exactly like we’re all tapped out. Probably not even of low hanging fruit.
There is evidence that the human brain is also doing "autocomplete" (prediction). The human brain uses predictive mechanisms when processing language, and these mechanisms play an important role in forming thoughts.
When we hear or read a word, our brain quickly generates a set of predictions about what word might come next, based on the context of the sentence and our past experiences with language. These predictions are constantly updated as we receive new information, and they help us to process language more efficiently and accurately.
In addition, research has shown that the brain engages in similar predictive processes when we are forming thoughts or planning actions. For example, when we plan a complex movement, such as reaching for a cup, our brain generates a set of predictions about the movements required to complete the action. These predictions are constantly updated as we receive feedback from our muscles and our environment, allowing us to make adjustments and achieve our goal.
> When we hear or read a word, our brain quickly generates a set of predictions about what word might come next, based on the context of the sentence
Yes a big part of it is prediction but the brain also does something else which LLMs by themselves completely eschew. The human brain imagines in pictures, creates and uses abstractions to refine understanding, studies things and produces new knowledge. When human brains study the goal to understand is different than LLMs.
It's not only language or some tasks - it's literally everything. Predictive Processing Theory proposes that our whole model is predicting future and only then confirming it through our input signals (eyes, ears etc). I highly recommend The Experience Machine by Andy Clark which explains and arguments this theory very convincingly to the point where I firmly believe it to be true.
This is of course sometimes true, we take shortcuts to minimize cognitive effort. However, when the situation warrants it we think more carefully about which words to use to achieve goals, or to reach correspondence with the situation at hand. Or we move more precisely and carefully to do something we haven't done before. I've no doubt that an LLM can approximate whatever model of language a person has from their life experience, but I don't think this type of model is capable of active coping, making judgments, or of having accountability to the world it's meant to operate in.
Andrew's distinction between associative vs logical thinking reminds me of two kinds of programming thinking. Half of my time while programming is spent churning out glue code, satisfying a type checker and test suite, and implementing the simple solution that turns out to work. The other half of my time is spent slowly thinking through why some simple solution is wrong, and step by step proceeding towards the correct but complex solution. The former phase is intuitive, the latter phase is scientific, where I hypothesize, test, and repeat.
Reading through the code-as-transcript afterwards it's unclear which bits of code required shallow associative vs deep rational thinking, pure autocomplete vs latent chain of thought.
Readit News
I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go.
Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they found is that there isn't really a limit to how good it can get. There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train. And it can go far beyond what the best human Go player can ever do. Even though very smart humans have spent their lifetimes deeply studying the game, and Alpha Zero had to learn everything from scratch.
One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move. And that model is obviously very sophisticated, and if you are going to say that it doesn't understand the game of Go, I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.
Likewise, with large language models, while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead. As Hinton says, there is no real limit to how sophisticated they can get. When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains. And the closer it gets, the more sophisticated model it needs. In the sense that Alpha Zero needs to "understand" the game of Go to play effectively, the large language model needs to understand "the world" to get better at predicting.
For an LLM, this is not even close to being the case. The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. And we certainly haven't figured out a "model" which would let a computer generate new and valid data that expands its understanding of the world beyond its inputs, so self-training is a non-starter for LLMs. What the LLM is "understanding", and what it is reinforced to "understand" is not the world but the format of texts, and while it may get very good at understanding the format of texts, that isn't equivalent to an understanding of the world.
No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts.
To the bird who can intuitively sense and use electromagnetic waves for motion and guidance, your model of the world is fundamentally incomplete.
There is a projection of the world in text. Moreover training on additional modalities is trivial for a transformer. That's all that matters.
I think if you're going to be strict about this, you have to defend against the point of view that the same 'ding an sich' problem applies to both LLMs and people. And also whether if you had a limit sequence of KL divergences, one from a person's POV of the world, and one from an LLM's POV of texts, what it is about how a person approaches better grasp of reality - and likewise their KL divergence approaches 0, in some sense implying that their world model is becoming the same as the distribution of the world - that can only apply to people.
It seems possible to me that there is probably a great deal of lurking anthropocentrism that humanity is going to start noticing more and more in ourselves in the coming years, probably in both the direction of AI and the direction of other animals as we start to understand both better
Deleted Comment
LLMs do not directly model the world; they train on and model what people write about the world. It is an AI model of a computed gestalt human model of the world, rather than a model of the world directly. If you ask it a question, it tells you what it models someone else (a gestalt of human writing) is most likely say. That in turn is strengthened if user interaction accepts it and corrected only if someone tells it something different.
If we were to define that as what "understanding" is, we would equivalently be saying that a human bullshit artist would have expert understanding if only they produced more believable bullshit. (They also just "try to sound like an expert".)
Likewise, I'm not convinced that we can measure its understanding just by identifying inaccuracies or measuring the difference between its answers and expert answers - There would be no difference between bluffing your way through the interview (relying on your interviewer's limitations in how they interrogate you) and acing the interview.
There seems to be a fundamental difference in levels of indirection. Where we "map the territory", LLMs "map the maps of the territory".
It can be an arbitrarily good approximation, and practically very useful, but it's a strong ontological step to say one thing "is" another just because it can be used like it.
This is true. But human brains don't directly model the world either, they form an internal model based on what comes in through their senses. Humans have the advantage of being more "multi-modal," but that doesn't mean that they get more information or better information.
Much of my "modeling of the world" comes from the fact that I've read a lot of text. But of course I haven't read even a tiny fraction of what GPT4 has.
That said, LLMs can already train on images, as GPT4-V does. And the image generators as well do this, it's just a matter of time before the two are fully integrated. Later we'll see a lot more training on video and sound, and it all being integrated into a single model.
Most arguments then descend into confusing the human knowledge embedded in a textbook with the human agency to apply the embedded knowledge. Software that extracts the knowledge from all textbooks has nothing to do with the human agency to use that knowledge.
I love chatGPT4 and had signed up in the first few hours it was released but I actually canceled my subscription yesterday. Part because of the bullshit with the company these past few days but also because it had just become a waste of time the past few months for me. I learned so much this year but I hit a wall that to make any progress I need to read the textbooks on the subjects I am interested in just like I had to this time last year before chatGPT.
We also shouldn't forget that children anthropomorphize toys and dolls quite naturally. It is entirely natural to anthropomorphize a LLM and especially when it is designed to pretend it is typing back a response like a human would. It is not bullshitting you though when it pretends to type back a response about how it doesn't actually understand what it is writing.
Deleted Comment
It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move (or moves that are better than others) and hence a function that goes from the state of the game to the best move. We just don't have a way to compute this function, but it exists. And that function doesn't need the concept of lookahead, that's just an intuitive way of how could find some of its values. Likewise ML algorithms don't necessarily need lookahead, they can just try to approximate that function with enough precision by exploiting patterns in it. And that's why we can still craft puzzles that some AIs can't solve but humans can, by exploiting edge cases in that function that the ML algorithm didn't notice but are solvable with understanding of the game.
The thing is though, does this really matter if eventually we won't be able to notice the difference?
Is there really a difference between the two? If a certain move shapes the opponent's remaining possible moves into a smaller subset, hasn't AlphaGo "looked ahead"? In other words, when humans strategize and predict what happens in the real world, aren't they doing the same thing?
I suppose you could argue that humans also include additional world models in their planning, but it's not clear to me that these models are missing and impossible for machine learning models to generate during training.
The rules of the game are deterministic, but you may be going a step too far with that claim.
Is the game deterministic when your opponent is non-deterministic?
Is there an optimal move for any board state given that various opponents have varying strategies? What may be the best move against one opponent may not be the best move against another opponent.
While I imagine alpha go does some brute force and some tree exploration, I think the main "intelligent" component of alpha go is the ability to recognize a "good" game state from a "bad" game state based on that moment in time, not any future plans or possibilities. That pattern recognition is all it has once its planning algorithm has reached the leaves of the trees. Correct me if I'm wrong, but I doubt alpha go has a neural net evaluating an entire tree of moves all at once to discover meta strategies like "the opponent focusing on this area" or "the opponent feeling on the back foot."
You can therefore imagine a pattern recognition algorithm so good that it is able to pick a move by only looking 1 move into the future, based solely on local stone densities and structures. Just play wherever improves the board state the most. It does not even need to "understand" that a game is being played.
> while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead.
So I don't think this statement is necessarily true. "Understanding" is a major achievement, but I don't think it requires planning. A computer can understand that 2+2=4 or where to play in tic-tac-toe without any "planning".
That said, there's probably not much special about the concept of planning either. If it's just simulating a tree of future possibilities and pruning it based on evaluation, then many algorithms have already achieved that.
There’s no limit to how sophisticated a model can get, but,
1. That’s a property shared with many architectures, and not really that interesting,
2. There are limits to the specific ways that we train models,
3. We care about the relative improvement that these models deliver, for a given investment of time and money.
From a mathematical perspective, you can just kind of keep multiplying the size of your model, and you can prove that it can represent arbitrary complicated structures (like, internal mental models of the world). That doesn’t mean that your training methods will produce those complicated structures.
With Go, I can see how the model itself can be used to generate new, useful training data. How such a technique could be applied to LLMs is less clear, and its benefits are more dubious.
So trying to make an AI that solves the entire problem before writing the first letter will likely not result in a good solution while also making it compute way too much since it solves the entire problem for every token generated. That is the kind of AI we know how to train so for now that is what we have to live with, but it isn't the kind of AI that would be efficient or smart.
I don’t think that’s a given. AlphaZero may have found an extremely high local optimum that isn’t the global optimum.
When playing only against itself, it won’t be able to get out of that local optimum, and when getting closer and closer to it even may ‘forget’ how to play against players that make moves that AplhaGo never would make, and that may be sufficient for a human to beat it (something like that happened with computer chess in the early years, where players would figure out which board positions computers were bad at, and try to get such positions on the board)
I think you have to keep letting it play against other good players (human or computer) that play differently to have it keep improving, and even then, there’s no guarantee it will find a global optimum.
LLMs do not have a "planning" module or simulator. There is no way the LLM can plan.
Could build a planning system into an LLM? Possibly and probably, but that is still open research. LeCunn is trying to figure out how to train them effectively. But even an LLM with a planning system does not make it AGI.
Some will argue that iteratively feeding the output embedding back into the input will retain the context but even in those cases it rapidly diverges or as we say "hallucinates"... still happens even with large input context windows. So there is still no planning here and no world model or understanding.
And
> When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains.
For example, it can reach 25% of accuracy and have an math limit of 26%, so "forever getting closer to 100% with time" would still result in a waste of even infinite resources
> it will continue to converge towards perfection
Then someone discovered a flaw that made it repeatably beatable by relative amateurs in a way that no human player would be
https://www.vice.com/en/article/v7v5xb/a-human-amateur-beat-...
Dead Comment
I disagree. You can think in terms of a system that doesn't involve predictions at all, but has the same or similar enough outcome.
So an action network just learns patterns. Just like a chess player can learn what positions look good without thinking ahead.
Is it just me or does this read like “here is my assumption about what you said, and now here is my passive aggressive judgement about that assumption”? If you’re not certain about what they mean by the word “understand”, I bet you could ask and they might explain it. Just a suggestion.
If I had to critique Hinton's claims, I would say his usage of the word "understand" can be vague and communicate assumptions because it's from an ontology used for reasoning about human reasoning, not this new alien form of reasoning which language models embody.
Especially since I don't think we know that much about how human intelligence actually works.
It is technically correct that when you take things to the extreme you can accomplish great things, but we may not reach those levels. We may require completely different technology to reach those levels of autocomplete, and we have simply reached a new plateau at this point in time.
We have reached those levels lol. That's why we're having this argument.
ChatGPT has a model of all the text information on the internet, but it remains to be seen what the hard limits of this model are. Does this model let it do logic or predict the future well, or will no amount of training give it those abilities? Simply being good in one task doesn't imply a general ability to do everything, or even most of everything. LLM's would simply be the last advancement in a field with a lot of similar advancements.
I have an idea that I want to convey, but how each word comes to my mind as I form a sentence has always felt like it's controlled by an unconscious algorithm.
So I don't understand why people find this prediction mechanism so alien.
It isn't clear to me how much of communication is really in our control.
With the current tools, it feels like we still provide the ideas we want the AI to convey, and it may be using a nearly identical mechanism to us to form the words.
Consciousness would be the computer being able to come up with the ideas.
So, it seems to me we've gotten close enough on the communication side of intelligence.
But the machine is not conscious. When it is, it seems like it will generate its own ideas.
Are people debating whether the machine is conscious?
Otherwise, it feels very straightforward to grasp what we've made up to now.
Why not describe things in terms of what they do instead of what they "are". The latter is highly subjective and open to abuse.
NB. By "things" I mean software and the type of vacuous companies discussed on HN, not people (a bizarre comparison). For example, websites that go on and on about some so-called "tech" copmany but never once tell the reader what the company does. Or silly memes like "It's X for Y". What does it do and how does it work are questions that often go unasked and unanswered.
A few days ago someone related a story of working for a company that produced some software it claimed used "AI" but according to the commenter it used nothing more than regular expressions. Was ELIZA "AI". Maybe we should ask what isn't "AI". What happens with "magic" if the audience knows how the trick is performed.
Would you say that about your spouse? The beauty of beholding one's wife is in who and what she is. What she does tells us something about who and what she is, to be sure, but any attempt to suppress the what (and the who) is dehumanizing and objectifying.
But, of course, what a thing does depends on what that thing is.
The reason I can say a human being can sort a list of numbers is because human beings have intention. When a human being sorts of list of numbers, they intend to sort the list. The intention is the cause and explanation for the actions taken the lead to a list of ordered numbers, as well as the resulting list of ordered numbers.
Does a computer sort numbers? In common speech, we say it does, just as we use all sorts of anthropomorphizing language when discussing computers. But at best, this is loose and analogical language. That's fine, as far as it goes, as long as we don't take it or mean it literally. However, the computer itself lacks intention. It is our intention that produces the computer, and our intention that makes the computer an instrument used by us to sort. Taken by itself, the computer is undergoing a transformation that effects something that we may interpreted as a list of sorted numbers, but the computer itself is not sorting. You wouldn't say that the clouds add x and y when x liters of water falls into a pool of y liters.
> The latter is highly subjective and open to abuse.
On the contrary, what a thing is is the most real and objective thing there is. An effect cannot be understood without knowing the cause, and the cause cannot be understood without knowing the agent. You can know some things about the effect, sure, and here the effect is that the text produced may be interpreted as intelligible. But the apparent intelligibility is borrowed from the source text, perhaps just a clever trick.
It seems to me that it's based on the magical thought that there's something truly special and unique about us humans, as compared to other species or technology. Those discussions always seem more theological than science-driven to me. If you want to measure the difference between human performance and LLMs, there's a million experiments you can run. I'll gladly be convinced by data, and I'm open to the possibility that the data might point in either direction, or be inconclusive. But grand words without data are not convincing.
Somehow OP seemed to twist that into “because I think on autopilot most of the time, then chatbots must think too”. Which is not totally incongruous with Hinton’s quote so much as a weird thing to balloon into an essay.
What's a little more concerning to me is that people are reading and upvoting it. I think, because I have hopes and aspirations about working on some very hard problems and communicating them to the public at some point. And if this is the level of 'ooh, squirrel'! that we're going at, that the work that I make might get overshadowed by something really silly.
Perhaps an odd insecurity, but there it is, I think.
Can you elaborate on this? I've studied some information theory and I don't see it.
When we hear or read a word, our brain quickly generates a set of predictions about what word might come next, based on the context of the sentence and our past experiences with language. These predictions are constantly updated as we receive new information, and they help us to process language more efficiently and accurately.
In addition, research has shown that the brain engages in similar predictive processes when we are forming thoughts or planning actions. For example, when we plan a complex movement, such as reaching for a cup, our brain generates a set of predictions about the movements required to complete the action. These predictions are constantly updated as we receive feedback from our muscles and our environment, allowing us to make adjustments and achieve our goal.
See links below for additional details:
https://www.earth.com/news/our-brains-are-constantly-working...
https://www.psycholinguistics.com/gerry_altmann/research/pap...
https://www.tandfonline.com/doi/pdf/10.1080/23273798.2020.18...
https://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2009...
Yes a big part of it is prediction but the brain also does something else which LLMs by themselves completely eschew. The human brain imagines in pictures, creates and uses abstractions to refine understanding, studies things and produces new knowledge. When human brains study the goal to understand is different than LLMs.
Reading through the code-as-transcript afterwards it's unclear which bits of code required shallow associative vs deep rational thinking, pure autocomplete vs latent chain of thought.
Deleted Comment