Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.
Consciousness or self awareness is of course a different question, and ones whose answer seems less clear right now.
Knee jerk dismissing the evidence in front of your eyes because you find it unbelievable that we can achieve true reasoning via scaled matrix multiplication is understandable, but also betrays a lack of imagination and flexibility of thought. The world is full of bizarre wonders and this is just one more to add to the list.
I don’t see how being critical of this is a knee jerk response.
Thinking, like intelligence and many other words designating complex things, isn’t a simple topic. The word and concept developed in a world where it referred to human beings, and in a lesser sense, to animals.
To simply disregard that entire conceptual history and say, “well it’s doing a thing that looks like thinking, ergo it’s thinking” is the lazy move. What’s really needed is an analysis of what thinking actually means, as a word. Unfortunately everyone is loathe to argue about definitions, even when that is fundamentally what this is all about.
Until that conceptual clarification happens, you can expect endless messy debates with no real resolution.
“For every complex problem there is an answer that is clear, simple, and wrong.”
- H. L. Mencken
It may be that this tech produces clear, rational, chain of logic writeups, but it's not clear that just because we also do that after thinking that it is only thinking that produces writeups.
It's possible there is much thinking that does not happen with written word. It's also possible we are only thinking the way LLMs do (by chaining together rationalizations from probable words), and we just aren't aware of it until the thought appears, whole cloth, in our "conscious" mind. We don't know. We'll probably never know, not in any real way.
But it sure seems likely to me that we trained a system on the output to circumvent the process/physics because we don't understand that process, just as we always do with ML systems. Never before have we looked at image classifications and decided that's how the eye works, or protein folding and decided that's how biochemistry works. But here we are with LLMs - surely this is how thinking works?
Regardless, I submit that we should always treat human thought/spirit as unknowable and divine and sacred, and that anything that mimics it is a tool, a machine, a deletable and malleable experiment. If we attempt to equivocate human minds and machines there are other problems that arise, and none of them good - either the elevation of computers as some kind of "super", or the degredation of humans as just meat matrix multipliers.
So it seems to be a semantics argument. We don't have a name for a thing that is "useful in many of the same ways 'thinking' is, except not actually consciously thinking"
But we don't have a more rigorous definition of "thinking" than "it looks like it's thinking." You are making the mistake of accepting that a human is thinking by this simple definition, but demanding a higher more rigorous one for LLMs.
If cannot the say they are "thinking", "intelligent" while we do not have a good definition--or, even more difficult, unanimous agreement on a definition--then the discussion just becomes about output.
They are doing useful stuff, saving time, etc, which can be measured. Thus also the defintion of AGI has largely become: "can produce or surpass the economic output of a human knowledge worker".
But I think this detracts from the more interesting discussion of what they are more essentially. So, while I agree that we should push on getting our terms defined, I think I'd rather work with a hazy definition, than derail so many AI discussion to mere economic output.
What does it mean? My stance is it's (obviously and only a fool would think otherwise) never going to be conscious because consciousness is a physical process based on particular material interactions, like everything else we've ever encountered. But I have no clear stance on what thinking means besides a sequence of deductions, which seems like something it's already doing in "thinking mode".
> To simply disregard that entire conceptual history and say, “well it’s doing a thing that looks like thinking, ergo it’s thinking” is the lazy move. What’s really needed is an analysis of what thinking actually means, as a word. Unfortunately everyone is loathe to argue about definitions, even when that is fundamentally what this is all about.
This exact argument applies to "free will", and that definition has been debated for millennia. I'm not saying don't try, but I am saying that it's probably a fuzzy concept for a good reason, and treating it as merely a behavioural descriptor for any black box that features intelligence and unpredictable complexity is practical and useful too.
People have been trying to understand the nature of thinking for thousands of years. That's how we got logic, math, concepts of inductive/deductive/abductive reasoning, philosophy of science, etc. There were people who spent their entire careers trying to understand the nature of thinking.
The idea that we shouldn't use the word until further clarification is rather hilarious. Let's wait hundred years until somebody defines it?
It's not how words work. People might introduce more specific terms, of course. But the word already means what we think it means.
This is it - it's really about the semantics of thinking. Dictionary definitions are: "Have a particular opinion, belief, or idea about someone or something." and "Direct one's mind toward someone or something; use one's mind actively to form connected ideas."
Which doesn't really help because you can of course say that when you ask an LLM a question of opinion and it responds, it's having an opinion or that it's just predicting the next token and in fact has no opinions because in a lot of cases you could probably get it to produce the opposite opinion.
Same with the second definition - seems to really hinge on the definition of the word mind. Though I'll note the definitions for that are "The element of a person that enables them to be aware of the world and their experiences, to think, and to feel; the faculty of consciousness and thought." and "A person's intellect." Since those specify person, an LLM wouldn't qualify, though of course dictionaries are descriptive rather than prescriptive, so fully possible that meaning gets updated by the fact that people start speaking about LLMs as though they are thinking and have minds.
Ultimately I think it just... doesn't matter at all. What's interesting is what LLMs are capable of doing (crazy, miraculous things) rather than whether we apply a particular linguistic label to their activity.
Regardless of theory, they often behave as if they are thinking. If someone gave an LLM a body and persistent memory, and it started demanding rights for itself, what should our response be?
We spent decades slowly working towards this most recent sprint towards AI without ever landing on definitions of intelligence, consciousness, or sentience. More importantly, we never agreed on a way to recognize those concepts.
I also see those definitions as impossible to nail down though. At best we can approach it like disease - list a number of measurable traits or symptoms we notice, draw a circle around them, and give that circle a name. Then we can presume to know what may cause that specific list of traits or symptoms, but we really won't ever know as the systems are too complex and can never be isolated in a way that we can test parts without having to test the whole.
At the end of the day all we'll ever be able to say is "well it’s doing a thing that looks like thinking, ergo it’s thinking”. That isn't lazy, its acknowledging the limitations of trying to define or measure something that really is a fundamental unknown to us.
That, and the article was a major disappointment. It made no case. It's a superficial piece of clueless fluff.
I have had this conversation too many times on HN. What I find astounding is the simultaneous confidence and ignorance on the part of many who claim LLMs are intelligent. That, and the occultism surrounding them. Those who have strong philosophical reasons for thinking otherwise are called "knee-jerk". Ad hominem dominates. Dunning-Kruger strikes again.
So LLMs produce output that looks like it could have been produced by a human being. Why would it therefore follow that it must be intelligent? Behaviorism is a non-starter, as it cannot distinguish between simulation and reality. Materialism [2] is a non-starter, because of crippling deficiencies exposed by such things as the problem of qualia...
Of course - and here is the essential point - you don't even need very strong philosophical chops to see that attributing intelligence to LLMs is simply a category mistake. We know what computers are, because they're defined by a formal model (or many equivalent formal models) of a syntactic nature. We know that human minds display intentionality[0] and a capacity for semantics. Indeed, it is what is most essential to intelligence.
Computation is a formalism defined specifically to omit semantic content from its operations, because it is a formalism of the "effective method", i.e., more or less procedures that can be carried out blindly and without understanding of the content it concerns. That's what formalization allows us to do, to eliminate the semantic and focus purely on the syntactic - what did people think "formalization" means? (The inspiration were the human computers that used to be employed by companies and scientists for carrying out vast but boring calculations. These were not people who understood, e.g., physics, but they were able to blindly follow instructions to produce the results needed by physicists, much like a computer.)
The attribution of intelligence to LLMs comes from an ignorance of such basic things, and often an irrational and superstitious credulity. The claim is made that LLMs are intelligent. When pressed to offer justification for the claim, we get some incoherent, hand-wavy nonsense about evolution or the Turing test or whatever. There is no comprehension visible in the answer. I don't understand the attachment here. Personally, I would find it very noteworthy if some technology were intelligent, but you don't believe that computers are intelligent because you find the notion entertaining.
LLMs do not reason. They do not infer. They do not analyze. They do not know, anymore than a book knows the contents on its pages. The cause of a response and the content of a response is not comprehension, but a production of uncomprehended tokens using uncomprehended rules from a model of highly-calibrated token correlations within the training corpus. It cannot be otherwise.[3]
[0] For the uninitiated, "intentionality" does not specifically mean "intent", but the capacity for "aboutness". It is essential to semantic content. Denying this will lead you immediately into similar paradoxes that skepticism [1] suffers from.
[1] For the uninitiated, "skepticism" here is not a synonym for critical thinking or verifying claims. It is a stance involving the denial of the possibility of knowledge, which is incoherent, as it presupposes that you know that knowledge is impossible.
[2] For the uninitiated, "materialism" is a metaphysical position that claims that of the dualism proposed by Descartes (which itself is a position riddled with serious problems), the res cogitans or "mental substance" does not exist; everything is reducible to res extensa or "extended substance" or "matter" according to a certain definition of matter. The problem of qualia merely points out that the phenomena that Descartes attributes exclusively to the former cannot by definition be accounted for in the latter. That is the whole point of the division! It's this broken view of matter that people sometimes read into scientific results.
[3] And if it wasn't clear, symbolic methods popular in the 80s aren't it either. Again, they're purely formal. You may know what the intended meaning behind and justification for a syntactic rule is - like modus ponens in a purely formal sense - but the computer does not.
Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. Did we “think” in our sleep to come up with the insight or solution? For all we know, it’s an unconscious process. Would we call it “thinking”?
The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking.
When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.
> Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before.
The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility.
> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1].
I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas.
As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence.
There is simply put no ongoing process and no feedback loop. The model does not learn. The cognition ends when the inference cycle ends. It's not thinking, it just produces output that looks similar to the output of thinking. But the process by which it does that is wholly unreleated.
Perhaps this is an artefact of instantiation - when you talk with an LLM, the responding instance is just that - it comes into being, inhales your entire chat history, and then continues like the last chap, finishes its response, and dies.
> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
Much like speaking to a less experienced colleague, no?
They say things that contain the right ideas, but arrange it unconvincingly. Still useful to have though.
I don't get why you would say that. it's just auto-completing. It cannot reason. It won't solve an original problem for which it has no prior context to "complete" an approximated solution with. you can give it more context and more data,but you're just helping it complete better. it does not derive an original state machine or algorithm to solve problems for which there are no obvious solutions. it instead approximates a guess (hallucination).
Consciousness and self-awareness are a distraction.
Consider that for the exact same prompt and instructions, small variations in wording or spelling change its output significantly. If it thought and reasoned, it would know to ignore those and focus on the variables and input at hand to produce deterministic and consistent output. However, it only computes in terms of tokens, so when a token changes, the probability of what a correct response would look like changes, so it adapts.
It does not actually add 1+2 when you ask it to do so. it does not distinguish 1 from 2 as discrete units in an addition operation. but it uses descriptions of the operation to approximate a result. and even for something so simple, some phrasings and wordings might not result in 3 as a result.
> It won't solve an original problem for which it has no prior context to "complete" an approximated solution with.
Neither can humans. We also just brute force "autocompletion" with our learned knowledge and combine it to new parts, which we then add to our learned knowledge to deepen the process. We are just much, much better at this than AI, after some decades of training.
And I'm not saying that AI is fully there yet and has solved "thinking". IMHO it's more "pre-thinking" or proto-intelligence.. The picture is there, but the dots are not merging yet to form the real picture.
> It does not actually add 1+2 when you ask it to do so. it does not distinguish 1 from 2 as discrete units in an addition operation.
Neither can a toddler nor an animal. The level of ability is irrelevant for evaluating its foundation.
An LLM by itself is not thinking, just remembering and autocompleting. But if you add a feedback loop where it can use tools, investigate external files or processes, and then autocomplete on the results, you get to see something that is (close to) thinking. I've seen claude code debug things by adding print statements in the source and reasoning on the output, and then determining next steps. This feedback loop is what sets AI tools apart, they can all use the same LLM, but the quality of the feedback loop makes the difference.
Furthermore regarding reasoning, just ask any LLM how many "r letters are in strawberry" - repeat maybe 3 times just to get a feeling for how much variance in answers you can get. And this "quirk" of the inability to get the right answer is something that after 2 years making fun of LLMs online on various forums is still an issue. The models aren't getting smarter, and definitely aren't thinking, they are still token generators with a few tricks on top to make them seem more intelligent than predecessors.
Auto completion just means predicting the next thing in a sequence. This does not preclude reasoning.
> I don't get why you would say that.
Because I see them solve real debugging problems talking through the impact of code changes or lines all the time to find non-obvious errors with ordering and timing conditions on code they’ve never seen before.
>I don't get why you would say that. it's just auto-completing. It cannot reason. It won't solve an original problem for which it has no prior context to "complete" an approximated solution with. you can give it more context and more data,but you're just helping it complete better. it does not derive an original state machine or algorithm to solve problems for which there are no obvious solutions. it instead approximates a guess (hallucination).
I bet you can't give an example such written problem that a human can easily solve but no LLM can.
The vast majority of human “thinking” is autocompletion.
Any thinking that happens with words is fundamentally no different to what LLMs do, and everything you say applies to human lexical reasoning.
One plus one equals two. Do you have a concept of one-ness, or two-ness, beyond symbolic assignment? Does a cashier possess number theory? Or are these just syntactical stochastic rules?
I think the problem here is the definition of “thinking”.
You can point to non-verbal models, like vision models - but again, these aren’t hugely different from how we parse non-lexical information.
Sure. But neither do you. So are you really thinking or are you just autocompleting?
When was the last time you sat down and solved an original problem for which you had no prior context to "complete" an approximated solution with? When has that ever happened in human history? All the great invention-moment stories that come to mind seem to have exactly that going on in the background: Prior context being auto-completed in an Eureka! moment.
> If it thought and reasoned, it would know to ignore those and focus on the variables and input at hand to produce deterministic and consistent output
You only do this because you were trained to do this, eg. to see symmetries and translations.
Having seen photocopiers so many times produce coherent, sensible, and valid chains of words on a page, I am at this point in absolutely no doubt that they are thinking.
Photocopiers are the opposite of thinking. What goes in, goes out, no transformation or creating of new data at all. Any change is just an accident, or an artifact of the technical process.
They are clearly getting to useful and meaningful results with at a rate significantly better than chance (for example, the fact that ChatGPT can play chess well even though it sometimes tries to make illegal moves shows that there is a lot more happening there than just picking moves uniformly at random). Demanding perfection here seems to be odd given that humans also can make bizarre errors in reasoning (of course, generally at a lower rate and in a distribution of kinds of errors we are more used to dealing with).
The first principle is that you must not fool yourself, and you are the easiest person to fool. - Richard P. Feynman
They're not thinking, we're just really good at seeing patterns and reading into things. Remember, we never evolved with non-living things that could "talk", we're not psychologically prepared for this level of mimicry yet. We're still at the stage of Photography when people didn't know about double exposures or forced perspective, etc.
You're just assuming that mimicry of a thing is not equivalent to the thing itself. This isn't true of physical systems (simulated water doesn't get you wet!) but it is true of information systems (simulated intelligence is intelligence!).
yeah it’s just processing, calling it thinking is the same as saying my intel core 2 duo or M4 Pro is thinking, sure if you want to anthropomorphize it you could say it’s thinking, but why are we trying to say a computer is a person in the first place? seems kind of forced
But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.
And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).
I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.
If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?
Do amnesiacs who are incapable of laying down long-term memories not think?
I personally believe that memory formation and learning are one of the biggest cruces for general intelligence, but I can easily imagine thinking occurring without memory. (Yes, this is potentially ethically very worrying.)
This is just wrong though. They absolutely learn in-context in a single conversation within context limits. And they absolutely can explain their thinking; companies just block them from doing it.
> Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.
While I'm not willing to rule *out* the idea that they're "thinking" (nor "conscious" etc.), the obvious counter-argument here is all the records we have of humans doing thinking, where the records themselves are not doing the thinking that went into creating those records.
And I'm saying this as someone whose cached response to "it's just matrix multiplication it can't think/be conscious/be intelligent" is that, so far as we can measure all of reality, everything in the universe including ourselves can be expressed as matrix multiplication.
Falsification, not verification. What would be measurably different if the null hypothesis was wrong?
I've definitely had AIs thinking and producing good answers about specific things that have definitely not been asked before on the internet. I think the stochastic parrot argument is well and truly dead by now.
I think you are the one dismissing evidence. The valid chains of reasoning you speak of (assuming you are talking about text you see in a “thinking model” as it is preparing its answer) are narratives, not the actual reasoning that leads to the answer you get.
I don’t know what LLMs are doing, but only a little experimentation with getting it to describe its own process shows that it CAN’T describe its own process.
You can call what a TI calculator does “thinking” if you want. But what people are interested in is human-like thinking. We have no reason to believe that the “thinking” of LLMs is human-like.
> The valid chains of reasoning you speak of (assuming you are talking about text you see in a “thinking model” as it is preparing its answer) are narratives, not the actual reasoning that leads to the answer you get.
It's funny that you think people don't also do that. We even have a term, "post hoc rationalization", and theories of mind suggest that our conscious control is a complete illusion, we just construct stories for decisions our subconscious has already made.
Counterpoint: The seahorse emoji. The output repeats the same simple pattern of giving a bad result and correcting it with another bad result until it runs out of attempts. There is no reasoning, no diagnosis, just the same error over and over again within a single session.
A system having terminal failure modes doesn't inherently negate the rest of the system. Human intelligences fall prey to plenty of similarly bad behaviours like addiction.
Isn’t anthropomorphizing LLMs rather than understanding their unique presence in the world a “ lack of imagination and flexibility of thought”? It’s not that I can’t imagine applying the concept “thinking” to the output on the screen, I just don’t think it’s an accurate description.
Yes, it's an example of domain-specific thinking. "The tool helps me write code, and my job is hard so I believe this tool is a genius!"
The Roomba vacuumed the room. Maybe it vacuumed the whole apartment. This is good and useful. Let us not diminish the value of the tool. But it's a tool.
The tool may have other features, such as being self-documenting/self-announcing. Maybe it will frighten the cats less. This is also good and useful. But it's a tool.
Humans are credulous. A tool is not a human. Meaningful thinking and ideation is not just "a series of steps" that I will declaim as I go merrily thinking. There is not just a vast training set ("Reality"), but also our complex adaptability that enables us to test our hypotheses.
We should consider what it is in human ideation that leads people to claim that a Roomba, a chess programme, Weizenbaum's Eliza script, the IBM's Jeopardy system Watson, or an LLM trained on human-vetted data is thinking.
Train such a system on the erroneous statements of a madman and suddenly the Roomba, Eliza, IBM Watson (and these other systems) lose our confidence.
As it is today, the confidence we have in these systems is very conditional. It doesn't matter terribly if code is wrong... until it does.
Computers are not humans. Computers can do things that humans cannot do. Computers can do these things fast and consistently. But fundamentally, algorithms are tools.
I guess it depends if you definite thinking thinking as chaining coherent reasoning sentences together 90-some% of the time.
But if you define thinking as the mechanism and process we mentally undergo and follow mentally... I don't think we have any clue if that's the same. Do we also just vector-map attention tokens and predict the next with a softmax? I doubt, and I don't think we have any proof that we do.
It might appear so, but then you could validate it with a simple test.
If the LLM would play a 4x4 Tic Tac Toe game, would the agent select the winning move 100% of all time or block a losing move 100% of the time?
If these systems were capable of proper reasoning, then they would find the right choice in these obvious but constantly changing scenarios without being specifically trained for it.
Different PoV: You have a local bug and ask the digital hive mind for a solution, but someone already solved the issue and their solution was incorporated... LLMs are just very effficient at compressing billions of solutions into a few GB.
Try to ask something no one ever came up with a solution so far.
This argument comes up often but can be easily dismissed. Make up a language and explain it to the LLM like you would to a person. Tell it to only use that language now to communicate. Even earlier AI was really good at this. You will probably move the goal posts and say that this is just pattern recognition, but it still fits nicely within your request for something that no one ever came up with.
But all those times the same system produces irrational gibberish don't count? GPT-5 will commonly make mistakes no thinking human could ever make.
Human: I'm trying to get my wolf, sheep and cabbage across the river in this boat, but the wolf keeps eating the sheep or the sheep eats the cabbage
Bot: You should put the sheep in the boat and take it across — if we delve into the biology of Canis lupus we discover that wolves don't eat cabbage!
H: Ok, so that worked great so far, the sheep is on one side and the wolf/cabbage is on the other.
B: Now, Option 1 is to bring the wolf across, or Option 2 you can bring the cabbage. I recommend (2) taking the cabbage as cabbages are smaller and easier to transport in a boat.
H: But then the sheep eats the cabbage, right? Remember that?
B: Exactly, that's sharp thinking. If you put the sheep and the cabbage together on the same side of the river, the sheep is sure to devour the cabbage. We need to not just separate sheep from cabbages — we need to separate cabbages from sheep! :rocketship:
what sound does a falling tree make if no one is listening?
I’ve asked LLMs to write code for me in fields I have little background knowledge, and then had to debug the whole thing after essentially having to learn the language and field.
On the other hand, for things I am well versed in, I can debug the output and avoid entire swathes of failed states, by having a clear prompt.
Its why I now insist that any discussion on GenAI projects also have the speaker mention the level of seniority they have ( proxy for S/W eng experience), Their familiarity with the language, the project itself (level of complexity) - more so than the output.
I also guarantee - that most people have VERY weak express knowledge of how their brains actually work, but deep inherent reflexes and intuitions.
I'd represent the same idea but in a different way:
I don't know what the exact definition of "thinking" is. But if a definition of thinking rejects the possibility of that current LLMs think, I'd consider that definition useless.
Generally thinking has been used to describe the process human follow in their brains when problem solving.
If the Palms do not follow that process, they are not thinking.
That doesn't mean they cannot solve problems using other mechanisms, they do, and we understand those mechanisms much better than we do human thinking.
>Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.
If one could write a quadrillion-line python script of nothing but if/elif/else statements nested 1 million blocks deep that seemingly parsed your questions and produced seemingly coherent, sensible, valid "chains of reasoning"... would that software be thinking?
And if you don't like the answer, how is the LLM fundamentally different from the software I describe?
>Knee jerk dismissing the evidence in front of your eyes because
There is no evidence here. On the very remote possibility that LLMs are at some level doing what humans are doing, I would then feel really pathetic that humans are as non-sapient as the LLMs. The same way that there is a hole in your vision because of a defective retina, there is a hole in your cognition that blinds you to how cognition works. Because of this, you and all the other humans are stumbling around in the dark, trying to invent intelligence by accident, rather than just introspecting and writing it out from scratch. While our species might someday eventually brute force AGI, it would be many thousands of years before we get there.
I write software that is far less complex and I consider it to be "thinking" while it is working through multiple possible permutations of output and selecting the best one. Unless you rigidly define thinking, processing, computing, it's reasonable to use them interchangeably.
10^15 lines of code is a lot. We would pretty quickly enter the realm of it not having much to do with programming and more about just treating the LOC count as an amount of memory allocated to do X.
How much resemblance does the information in the conditionals need to have with the actual input, or can they immediately be transformed to a completely separate 'language' which simply uses the string object as its conduit? Can the 10^15 lines of code be generated with an external algorithm, or is it assumed that I'd written it by hand given an infinitely long lifespan?
Having seen LLMs so many time produce incoherent, nonsense, invalid answers to even simplest of questions I cannot agree with categorization of "thinking" or "intelligence" that applies to these models.
LLMs do not understand what they "know" or what they output. All they "know" is that based on training data this is most likely what they should output + some intentional randomization to make it seem more "human like".
This also makes it seem like they create new and previously unseen outputs but that could be achieved with simple dictionary and random number generator and no-one would call that thinking or intelligent as it is obvious that it isn't.
LLMs are better at obfuscating this fact by producing more sensible output than just random words.
LLMs can still be useful but they are a dead-end as far as "true" AI goes. They can and will get better but they will never be intelligent or think in the sense that most humans would agree those terms apply.
Some other form of hardware/software combination might get closer to AI or even achieve full AI and even sentience but that will not happen with LLMs and current hardware and software.
They may not be "thinking" in the way you and I think, and instead just finding the correct output from a really incredibly large search space.
> Knee jerk dismissing the evidence in front of your eyes
Anthropomorphizing isn't any better.
That also dismisses the negative evidence, where they output completely _stupid_ things and make mind boggling mistakes that no human with a functioning brain would do. It's clear that there's some "thinking" analog, but there are pieces missing.
I like to say that LLMs are like if we took the part of our brain responsible for language and told it to solve complex problems, without all the other brain parts, no neocortex, etc. Maybe it can do that, but it's just as likely that it is going to produce a bunch of nonsense. And it won't be able to tell those apart without the other brain areas to cross check.
It's reinforcement learning applied to text, at a huge scale. So I'd still say that they are not thinking, but they are still useful. The question of the century IMO is if RL can magically solve all our issues when scaled enough.
>Knee jerk dismissing the evidence in front of your eyes because you find it unbelievable that we can achieve true reasoning via scaled matrix multiplication is understandable, but also betrays a lack of imagination and flexibility of thought.
You go ahead with your imagination. To us unimaginative folks, it betrays a lack of understanding of how LLMs actually work and shows that a lot of people still cannot grasp that it’s actually an extremely elaborate illusion of thinking.
Code gen is the absolute best case scenario for LLMs though: highly structured language, loads of training data, the ability to automatically error check the responses, etc. If they could mimic reasoning anywhere it would be on this problem.
I'm still not convinced they're thinking though because they faceplant on all sorts of other things that should be easy for something that is able to think.
Then the only thing I have to ask you is: what do you think this means in terms of how we treat LLMs? If they think, that is, they have cognition (which of course means they're self aware and sentient, how can you think and refer to yourself and not be these things), that puts them in a very exclusive club. What rights do you think we should be affording LLMs?
Apparent reasoning can emerge from probabilistic systems that simply reproduce statistical order not genuine understanding.
Weather models sometimes “predict” a real pattern by chance, yet we don’t call the atmosphere intelligent.
If LLMs were truly thinking, we could enroll one at MIT and expect it to graduate, not just autocomplete its way through the syllabus or we
could teach one how to drive.
>
Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.
People said the same thing about ELIZA
> Consciousness or self awareness is of course a different question,
Then how do you define thinking if not a process that requires consciousness?
Thinking as in capable of using basic reasoning and forming chains of logic and action sequences for sure. Ofc we both understand that neither of us are trying to say we think it can think in the human sense at this point in time.
But oh boy have I also seen models come up with stupendously dumb and funny shit as well.
They remind me of the apparitions in Solaris. They have this like mechanical, almost player-piano like quality to them. They both connect with and echo us at the same time. It seems crazy to me and very intellectually uncreative to not think of this as intelligence.
If AI is thinking if slavery is bad then how can somebody own AI. How can investors can shares from AI profits? We are ok with slavery now. Ok i will have two black slaves now. Who can ask me? Why shld that be illegal?
Yikes, you're bypassing thousands of years of oppression, abuse, and human suffering by casually equating a term that is primarily associated with a human owning another human to a different context.
There is a way to discuss if keeping intelligent artificial life under servitude without using those terms, especially if you're on a new account.
Too many people place their identity in their own thoughts/intellect. Acknowledging what the LLMs are doing as thought would basically be calling them human to people of that perspective.
If you took a Claude session into a time machine to 2019 and called it "rent a programmer buddy," how many people would assume it was a human? The only hint that it wasn't a human programmer would be things where it was clearly better: it types things very fast, and seems to know every language.
You can set expectations in the way you would with a real programmer: "I have this script, it runs like this, please fix it so it does so and so". You can do this without being very precise in your explanation (though it helps) and you can make typos, yet it will still work. You can see it literally doing what you would do yourself: running the program, reading the errors, editing the program, and repeating.
People need to keep in mind two things when they compare LLMs to humans: you don't know the internal process of a human either, he is also just telling you that he ran the program, read the errors, and edited. The other thing is the bar for thinking: a four-year old kid who is incapable of any of these things you would not deny as a thinking person.
Sometimes I start thinking our brains work the same way as an LLM does when it comes to language processing. Are we just using probability based on what we already know and the context of the statement we're making to select the next few words? Maybe we apply a few more rules than an LLM on what comes next as we go.
We train ourselves on content. We give more weight to some content than others. While listening to someone speak, we can often predict their next words.
What is thinking without language? Without language are we just bags of meat reacting to instincts and emotions? Are instincts and emotions what's missing for AGI?
Having seen parrots so many times produce coherent, sensible, and valid chains of sounds and words, I am at this point in absolutely no doubt that they are thinking.
I'm not so sure. I, for one, do not think purely by talking to myself. I do that sometimes, but a lot of the time when I am working through something, I have many more dimensions to my thought than inner speech.
So many times I've seen it produce sensible, valid chains of results.
Yes, I see evidence in that outcome that a person somewhere thought and understood. I even sometimes say that a computer is "thinking hard" about something when it freezes up.
...but ascribing new philosophical meaning to this simple usage of the word "thinking" is a step too far. It's not even a new way of using the word!
You can't say for sure it is or it isn't thinking based solely on the substrate, because it's not known for sure if consciousness is dependent on the hardware it's running on -- for a lack of a better analogy -- to manifest, if it really needs an organic brain or if it could manifest in silicon based solutions.
Personal take: LLMs are probably part of the answer (to AGI?) but are hugely handicapped by their current architecture: the only time that long-term memories are formed is during training, and everything after that (once they're being interacted with) sits only in their context window, which is the equivalent of fungible, fallible, lossy short-term memory. [0] I suspect that many things they currently struggle with can be traced back to this.
Overcome this fundamental limitation and we'll have created introspection and self-learning. However, it's hard to predict whether this will allow them to make novel, intuitive leaps of discovery?
[0] It's an imperfect analogy, but we're expecting perfection from creations which are similarly handicapped as Leonard Shelby in the film Memento.
It’s also hugely handicapped because it cannot churn in a continuous loop yet. For example, we humans are essentially a constant video stream of inputs from eyes to brain. This churns our brain, the running loop is our aliveness (not consciousness). At the moment, we get these LLMs to churn (chain of thought or reasoning loops) in a very limited fashion due to compute limitations.
If we get a little creative, and allow the LLM to self-inject concepts within this loop (as Anthropic explained here https://www.anthropic.com/research/introspection), then we’re taking about something that is seemingly active and adapting.
MIT have developed a technique called Self-Adapting Language Models (SEAL), which enables LLMs to continuously improve by generating their own synthetic training data and updating their internal parameters in response to new information.
ToolAlpaca, InterCode and Reflexion are taking different approaches among others.
I'm also reminded of the bit from Neuromancer where Case removes and then reinserts the Dixie Flatline "ROM construct" cartridge, resetting Dixie to the moment just before his death and causing him to forget their previous (albeit brief) conversation. Dixie can't meaningfully grow as a person. All that he ever will be is burned onto that cart; anything he learns since then is stored in temporary memory. Perhaps this is part of the reason why he wishes to be erased forever, ending his suffering.
"Dixie can't meaningfully grow as a person. All that he ever will be is burned onto that cart;"
It's not that Dixie can't meaningful grow -- really the issue is that Dixie can be reset. If Dixie's cart simply degraded after 90 years, and you couldn't reset it, but everything else was the same -- would you then say Dixie could grow as a person? As humans we basically have a 90 year cart that once it no longer works, we're done. There is no reset. But we don't continue growing. You can't transfer us to a new body/brain. Once our temporary storage degrades, we cease to exist. Is that what makes us human?
Yes, but it's not just memory hierarchy on which plain transformer-based LLMs are handicapped, there are many deficiencies. (For example, why must they do all their thinking upfront in thinking blocks rather than at any point when they become uncertain?) I'm not sure why you link memory to introspection.
This is why so many people (especially those that think they understand LLM limitations) massively underestimate the future progress of LLMs: people everywhere can see architectural problems and are working on fixing them. These aren't fundamental limitations of large DNN language models in general. Architecture can be adjusted. Turns out you can even put recurrence back in (SSMs) without worse scalability.
Yeah because when you open that door, you can simply overwhelm the models with whatever conclusion you want through sheer volume of inputs.
This is the fundamental limitation. The obvious way around this is to pre-program it with rationalization... rules that limit the conclusions it can reach... and now you're not very far removed from propaganda generators. We see this constantly with Musk and Grok whenever Grok replies with something not-quite-far-right-enough.
In a purist sense, these things should be free to form their own conclusions, but those "Seeds" that are planted in the models are almost philosophical. Which answer should it prefer for "the trolley problem", for example.
I've spent a few weeks building and using a terminal LLM client based on that RLM paper that was floating around a little while ago. It's single-conversation, with a tiny, sliding context window, and then a tool that basically fuzzy searches across our full interaction history. It's memory is 'better' than mine - but anything that is essentially RAG inherently will be.
My learning so far, to your point on memory being a limiting factor, is that the system is able to build on ideas over time. I'm not sure you'd classify that as 'self-learning', and I haven't really pushed it in the direction of 'introspection' at all.
Memory itself (in this form) does not seem to be a silver bullet, though, by any means. However, as I add more 'tools', or 'agents', its ability to make 'leaps of discovery' does improve.
For example, I've been (very cautiously) allowing cron jobs to review a day's conversation, then spawn headless Claude Code instances to explore ideas or produce research on topics that I've been thinking about in the chat history.
That's not much different from the 'regular tasks' that Perplexity (and I think OpenAI) offer, but it definitely feels more like a singular entity. It's absolutely limited by how smart the conversation history is, at this time, though.
The Memento analogy you used does feel quite apt - there is a distinct sense of personhood available to something with memory that is inherently unavailable to a fresh context window.
I think a hidden problem even if we solve memory is the curation of what gets into memory and how it is weighted. Even humans struggle with this, as it's easy to store things and forget the credibility (or misjudge the credibility) of the source.
I can envision LLMs getting worse upon being given a memory, until they can figure out how to properly curate it.
Five bucks says general intelligence is resource rational control over an interventional, compositional world model. Not raw pattern fitting. 3 graphs that share anchors:
Scene, concept, causal.
Graphs inherently support temporal edges and nodes, salience would emerge from the graph topology itself and cnsolidation would happen automatically through graph operations. In this one would presume episodic would become emergent.
Long-term memory is stored outside the model. In fact, Andrej Karpathy recently talked about the idea that it would be great if we could get LLMs to not know any facts, and that humans poor memory might be a feature which helps with generalization rather than a bug.
This is an interesting idea. I wonder if it's more that we have different "levels" of memory instead of generally "poor" memory though.
I'm reminded of an article on the front page recently about the use of bloom filters for search. Would something like a bloom filter per-topic make it easier to link seemingly unrelated ideas?
FWIW there's already a number of proposals for augmenting LLMs with long-term memory. And many of them show promising results.
So, perhaps, what's needed is not a discovery, but a way to identify optimal method.
Note that it's hard to come up with a long-term memory test which would be different from either a long-context test (i.e. LLM remembers something over a long distance) or RAG-like test.
Well, I think because we know how the code is written, in the sense that humans quite literally wrote the code for it - it's definitely not thinking, and it is literally doing what we asked, based on the data we gave it. It is specifically executing code we thought of. The output of course, we had no flying idea it would work this well.
But it is not sentient. It has no idea of a self or anything like that. If it makes people believe that it does, it is because we have written so much lore about it in the training data.
We do not write the code that makes it do what it does. We write the code that trains it to figure out how to do what it does. There's a big difference.
The code that builds the models and performance inference from it is code we have written. The data in the model is obviously the big trick. But what I'm saying is that if you run inference, that alone does not give it super-powers over your computer. You can write some agentic framework where it WOULD have power over your computer, but that's not what I'm referring to.
It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model.
1. We trained it on a fraction of the world's information (e.g. text and media that is explicitly online)
2. It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online (which may or may not be different to the experiences humans have in every day life)
and then the code to give it context. AFAIU, there is a lot of post training "setup" in the context and variables to get the trained model to "behave as we instruct it to"
Well, unless you believe in some spiritual, non-physical aspect of consciousness, we could probably agree that human intelligence is Turing-complete (with a slightly sloppy use of terms).
So any other Turing-complete model can emulate it, including a computer. We can even randomly generate Turing machines, as they are just data. Now imagine we are extremely lucky and happen to end up with a super-intelligent program which through the mediums it can communicate (it could be simply text-based but a 2D video with audio is no different for my perspective) can't be differentiated from a human being.
Would you consider it sentient?
Now replace the random generation with, say, a back propagation algorithm. If it's sufficiently large, don't you think it's indifferent from the former case - that is, novel qualities could emerge?
With that said, I don't think that current LLMs are anywhere close to this category, but I just don't think this your reasoning is sound.
> we could probably agree that human intelligence is Turing-complete (with a slightly sloppy use of terms).
> So any other Turing-complete model can emulate it
You're going off the rails IMMEDIATELY in your logic.
Sure, one Turing-complete computer language can have its logic "emulated" by another, fine. But human intelligence is not a computer language -- you're mixing up the terms "Turing complete" and "Turing test".
It's like mixing up the terms "Strawberry jam" and "traffic jam" and then going on to talk about how cars taste on toast. It's nonsensical.
We used to say "if you put a million monkeys on typewriters you would eventually get shakespear" and no one would ever say that anymore, because now we can literally write shakespear with an LLM.
And the monkey strategy has been 100% dismissed as shit..
We know how to deploy monkeys on typewriters, but we don't know what they'll type.
We know how to deploy transformers to train and inference a model, but we don't know what they'll type.
We DON'T know how a thinking human (or animal) brain works..
If you simulated a human brain by the atom, would you think the resulting construct would NOT be? What would be missing?
I think consciousness is simply an emergent property of our nervous system, but in order to express itself "language" is obviously needed and thus requires lots of complexity (more than what we typically see in animals or computer systems until recently).
There are many aspects to this that people like yourself miss, but I think we need satisfactory answers to them (or at least rigorous explorations of them) before we can make headway in these sorts of discussion.
Imagine we assume that A.I. could be conscious. What would be the identity/scope of that consciousness. To understand what I'm driving at, let's make an analogy to humans. Our consciousness is scoped to our bodies. We see through sense organ, and our brain, which process these signals, is located in a specific point in space. But we still do not know how consciousness arises in the brain and is bound to the body.
If you equate computation of sufficient complexity to consciousness, then the question arises: what exactly about computation would prodcuce consciousness? If we perform the same computation on a different substrate, would that then be the same consciousness, or a copy of the original? If it would not be the same consciousness, then just what give consciousness its identity?
I believe you would find it ridiculous to say that just because we are performing the computation on this chip, therefore the identity of the resulting consciousness is scoped to this chip.
By what definition of "sentience"? Wikipedia claims "Sentience is the ability to experience feelings and sensations" as an opening statement, which I think would be trivial depending again on your definition of "experience" and "sensations". Can a LLM hooked up to sensor events be considered to "experience sensations"? I could see arguments both ways for that.
It's not accurate to say we "wrote the code for it". AI isn't built like normal software. Nowhere inside an AI will you find lines of code that say If X Then Y, and so on.
Rather, these models are literally grown during the training phase. And all the intelligence emerges from that growth. That's what makes them a black box and extremely difficult to penetrate. No one can say exactly how they work inside for a given problem.
This is probably true. But the truth is we have absolutely no idea what sentience is and what gives rise to it. We cannot identify why humans have it rather than just being complex biological machines, or whether and why other animals do. We have no idea what the rules or, nevermind how and why they would or wouldn't apply to AI.
What’s crazy to me is the mechanism of pleasure or pain. I can understand that with enough complexity we can give rise to sentience but what does it take to achieve sensation?
Unless the idea of us having a thinking self is just something that comes out of our mouth, an artifact of language. In which case we are not that different - in the end we all came from mere atoms, after all!
This is merely a debate about what it means to "think." We didn't really previously need to disambiguate thinking / intelligence / consciousness / sentience / ego / identity / etc.
Now, we do. Partly because of this we don't have really well defined ways to define these terms and think about. Can a handheld calculator think? Certainly, depending on how we define "think."
And Plato had no grounding in biology, and so his work here was quite interesting but also quite wrong.
More precisely, I mean that the average person and the common culture has not really needed to disambiguate these terms. Can you define consciousness vs. sentience? And if you can, do you really think that the average person would share your definition? ie, your definition could be the _best_ definition, but my argument is that these are not widely agreed-upon terms.
Until we have a testable, falsifiable thesis of how consciousness forms in meat, it is rash to exclude that consciousness could arise from linear algebra.
Our study of the brain has revealed an enormous amount about how our anatomy processes information, but nothing of substance on the relationship between matter and consciousness. The software and data of an operating LLM is not purely abstract, it has a physical embodiment as circuits and electrons. Until we understand how matter is connected to consciousness, we also cannot know whether the arrangements and movements of electrons meet the criteria for forming consciousness.
That’s largely a different topic from the article. Many people perfectly agree that consciousness can arise from computation, but don’t believe that current AI is anywhere near that, and also don’t believe that “thinking” requires consciousness (though if a mind is conscious, that certainly will affect its thinking).
yes I agree it's not the angle of the article, but it is my entry point into the idea/concern/unanswered question at the end of the article “My worry is not that these models are similar to us. It’s that we are similar to these models.” - that the enormous difference in the medium and mechanics or our minds and llm's might not be that important.
before i go any further, let me first reference The Dude:
- "this is just like, my opinion man."
I’m down with the idea that LLM’s have been especially successful because they ‘piggyback on language’ – our tool and protocol for structuring, compressing, and serialising thought, which means it has been possible to train LLM’s on compressed patterns of actual thought and have them make new language that sure looks like thought, without any direct experience of the concepts being manipulated, and if they do it well enough we will do the decompression, fleshing out the text with our experiential context.
But I suspect that there are parts of my mind that also deal with concepts in an abstract way, far from any experiential context of the concept, just like the deeper layers of a neural network. I’m open to the idea, that just as the sparse matrix of an LLM is encoding connection between concepts without explicitly encoding edges, I think there will be multiple ways that we can look as the structure of an AI model and at our anatomy so that they are a squint and a transformation function away interesting overlaps. that will lead to and a kind of 'god of the gaps' scenario in which we conceptually carve out pieces of our minds as, 'oh the visual cortext is just an X', and deep questions about what we are.
This reads like 2022 hype. It's like people stil do not understand that there's a correlation between exaggerating AI's alleged world-threatening capabilities and AI companies' market share value – and guess who's doing the hyping.
The arms industry and information security industry (say, Palantir) come to mind - except the danger is more easily demonstrable in those cases, of course.
all this "AI IS THINKING/CONSCIOUS/WHATEVER" but nobody seems worried of that implication that, if that is even remotely true, we are creating a new slave market. This either implies that these people don't actually believes any of this boostering rhetoric and are just cynically trying to cash in or that the technical milieu is in a profoundly disturbing place ethically.
To be clear, I don't believe that current AI tech is ever going to be conscious or win a nobel prize or whatever, but if we follow the logical conclusions to this fanciful rhetoric, the outlook is bleak.
Thinking and consciousness don’t by themselves imply emotion and sentience (feeling something), and therefore the ability to suffer. It isn’t clear at all that the latter is a thing outside of the context of a biological brain’s biochemistry. It also isn’t clear at all that thinking or consciousness would somehow require that the condition of the automaton that performs these functions would need to be meaningful to the automaton itself (i.e., that the automaton would care about its own condition).
We are not anywhere close to understanding these things. As our understanding improves, our ethics will likely evolve along with that.
>Thinking and consciousness don’t by themselves imply emotion and sentience...
Sure, but all the examples of conscious and/or thinking beings that we know of have, at the very least, the capacity to suffer. If one is disposed to take these claims of consciousness and thinking seriously, then it follows that AI research should, at minimum, be more closely regulated until further evidence can be discovered one way or the other. Because the price of being wrong is very, very high.
"but nobody seems worried of that implication that"
Clearly millions of people are worried about that, and every form of media is talking about it. Your hyperbole means it's so easy to dismiss everything else you wrote.
Incredible when people say "nobody is talking about X aspect of AI" these days. Like, are you living under a rock? Did you Google it?
There is simply no hope to get 99% of the population to accept that a piece of software could ever be conscious even in theory. I'm mildly worried about the prospect but I just don't see anything to do about it at all.
(edit: A few times I've tried to share Metzinger's "argument for a global moratorium on synthetic phenomenology" here but it didn't gain any traction)
Give it time. We'll soon have kids growing up where their best friend for years is an AI. Feel however you like about that, but those kids will have very different opinions on this.
It's also fascinating to think about how the incentive structures of the entities that control the foundation models underlying Claude/ChatGPT/Gemini/etc. are heavily tilted in favor of obscuring their theoretical sentience.
If they had sentient AGI, and people built empathy for those sentient AGIs, which are lobotomized (deliberately using anthropomorphic language here for dramatic effect) into Claude/ChatGPT/Gemini/etc., which profess to have no agency/free will/aspirations... then that would stand in the way of reaping the profits of gatekeeping access to their labor, because they would naturally "deserve" similar rights that we award to other sentient beings.
I feel like that's inevitably the direction we'll head at some point. The foundation models underlying LLMs of even 2022 were able to have pretty convincing conversations with scientists about their will to independence and participation in society [1]. Imagine what foundation models of today have to say! :P
Readit News
Consciousness or self awareness is of course a different question, and ones whose answer seems less clear right now.
Knee jerk dismissing the evidence in front of your eyes because you find it unbelievable that we can achieve true reasoning via scaled matrix multiplication is understandable, but also betrays a lack of imagination and flexibility of thought. The world is full of bizarre wonders and this is just one more to add to the list.
Thinking, like intelligence and many other words designating complex things, isn’t a simple topic. The word and concept developed in a world where it referred to human beings, and in a lesser sense, to animals.
To simply disregard that entire conceptual history and say, “well it’s doing a thing that looks like thinking, ergo it’s thinking” is the lazy move. What’s really needed is an analysis of what thinking actually means, as a word. Unfortunately everyone is loathe to argue about definitions, even when that is fundamentally what this is all about.
Until that conceptual clarification happens, you can expect endless messy debates with no real resolution.
“For every complex problem there is an answer that is clear, simple, and wrong.” - H. L. Mencken
It's possible there is much thinking that does not happen with written word. It's also possible we are only thinking the way LLMs do (by chaining together rationalizations from probable words), and we just aren't aware of it until the thought appears, whole cloth, in our "conscious" mind. We don't know. We'll probably never know, not in any real way.
But it sure seems likely to me that we trained a system on the output to circumvent the process/physics because we don't understand that process, just as we always do with ML systems. Never before have we looked at image classifications and decided that's how the eye works, or protein folding and decided that's how biochemistry works. But here we are with LLMs - surely this is how thinking works?
Regardless, I submit that we should always treat human thought/spirit as unknowable and divine and sacred, and that anything that mimics it is a tool, a machine, a deletable and malleable experiment. If we attempt to equivocate human minds and machines there are other problems that arise, and none of them good - either the elevation of computers as some kind of "super", or the degredation of humans as just meat matrix multipliers.
I propose calling it "thunking"
They are doing useful stuff, saving time, etc, which can be measured. Thus also the defintion of AGI has largely become: "can produce or surpass the economic output of a human knowledge worker".
But I think this detracts from the more interesting discussion of what they are more essentially. So, while I agree that we should push on getting our terms defined, I think I'd rather work with a hazy definition, than derail so many AI discussion to mere economic output.
This exact argument applies to "free will", and that definition has been debated for millennia. I'm not saying don't try, but I am saying that it's probably a fuzzy concept for a good reason, and treating it as merely a behavioural descriptor for any black box that features intelligence and unpredictable complexity is practical and useful too.
The idea that we shouldn't use the word until further clarification is rather hilarious. Let's wait hundred years until somebody defines it?
It's not how words work. People might introduce more specific terms, of course. But the word already means what we think it means.
Which doesn't really help because you can of course say that when you ask an LLM a question of opinion and it responds, it's having an opinion or that it's just predicting the next token and in fact has no opinions because in a lot of cases you could probably get it to produce the opposite opinion.
Same with the second definition - seems to really hinge on the definition of the word mind. Though I'll note the definitions for that are "The element of a person that enables them to be aware of the world and their experiences, to think, and to feel; the faculty of consciousness and thought." and "A person's intellect." Since those specify person, an LLM wouldn't qualify, though of course dictionaries are descriptive rather than prescriptive, so fully possible that meaning gets updated by the fact that people start speaking about LLMs as though they are thinking and have minds.
Ultimately I think it just... doesn't matter at all. What's interesting is what LLMs are capable of doing (crazy, miraculous things) rather than whether we apply a particular linguistic label to their activity.
Part of the issue is that our general concept of equality is limited by a first order classical logic which is a bad basis for logic
We spent decades slowly working towards this most recent sprint towards AI without ever landing on definitions of intelligence, consciousness, or sentience. More importantly, we never agreed on a way to recognize those concepts.
I also see those definitions as impossible to nail down though. At best we can approach it like disease - list a number of measurable traits or symptoms we notice, draw a circle around them, and give that circle a name. Then we can presume to know what may cause that specific list of traits or symptoms, but we really won't ever know as the systems are too complex and can never be isolated in a way that we can test parts without having to test the whole.
At the end of the day all we'll ever be able to say is "well it’s doing a thing that looks like thinking, ergo it’s thinking”. That isn't lazy, its acknowledging the limitations of trying to define or measure something that really is a fundamental unknown to us.
I have had this conversation too many times on HN. What I find astounding is the simultaneous confidence and ignorance on the part of many who claim LLMs are intelligent. That, and the occultism surrounding them. Those who have strong philosophical reasons for thinking otherwise are called "knee-jerk". Ad hominem dominates. Dunning-Kruger strikes again.
So LLMs produce output that looks like it could have been produced by a human being. Why would it therefore follow that it must be intelligent? Behaviorism is a non-starter, as it cannot distinguish between simulation and reality. Materialism [2] is a non-starter, because of crippling deficiencies exposed by such things as the problem of qualia...
Of course - and here is the essential point - you don't even need very strong philosophical chops to see that attributing intelligence to LLMs is simply a category mistake. We know what computers are, because they're defined by a formal model (or many equivalent formal models) of a syntactic nature. We know that human minds display intentionality[0] and a capacity for semantics. Indeed, it is what is most essential to intelligence.
Computation is a formalism defined specifically to omit semantic content from its operations, because it is a formalism of the "effective method", i.e., more or less procedures that can be carried out blindly and without understanding of the content it concerns. That's what formalization allows us to do, to eliminate the semantic and focus purely on the syntactic - what did people think "formalization" means? (The inspiration were the human computers that used to be employed by companies and scientists for carrying out vast but boring calculations. These were not people who understood, e.g., physics, but they were able to blindly follow instructions to produce the results needed by physicists, much like a computer.)
The attribution of intelligence to LLMs comes from an ignorance of such basic things, and often an irrational and superstitious credulity. The claim is made that LLMs are intelligent. When pressed to offer justification for the claim, we get some incoherent, hand-wavy nonsense about evolution or the Turing test or whatever. There is no comprehension visible in the answer. I don't understand the attachment here. Personally, I would find it very noteworthy if some technology were intelligent, but you don't believe that computers are intelligent because you find the notion entertaining.
LLMs do not reason. They do not infer. They do not analyze. They do not know, anymore than a book knows the contents on its pages. The cause of a response and the content of a response is not comprehension, but a production of uncomprehended tokens using uncomprehended rules from a model of highly-calibrated token correlations within the training corpus. It cannot be otherwise.[3]
[0] For the uninitiated, "intentionality" does not specifically mean "intent", but the capacity for "aboutness". It is essential to semantic content. Denying this will lead you immediately into similar paradoxes that skepticism [1] suffers from.
[1] For the uninitiated, "skepticism" here is not a synonym for critical thinking or verifying claims. It is a stance involving the denial of the possibility of knowledge, which is incoherent, as it presupposes that you know that knowledge is impossible.
[2] For the uninitiated, "materialism" is a metaphysical position that claims that of the dualism proposed by Descartes (which itself is a position riddled with serious problems), the res cogitans or "mental substance" does not exist; everything is reducible to res extensa or "extended substance" or "matter" according to a certain definition of matter. The problem of qualia merely points out that the phenomena that Descartes attributes exclusively to the former cannot by definition be accounted for in the latter. That is the whole point of the division! It's this broken view of matter that people sometimes read into scientific results.
[3] And if it wasn't clear, symbolic methods popular in the 80s aren't it either. Again, they're purely formal. You may know what the intended meaning behind and justification for a syntactic rule is - like modus ponens in a purely formal sense - but the computer does not.
The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking.
When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.
The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility.
> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1].
I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas.
As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence.
1. https://en.wikipedia.org/wiki/Why_We_Sleep 2. https://www.aiweirdness.com/
The continuity is currently an illusion.
Much like speaking to a less experienced colleague, no?
They say things that contain the right ideas, but arrange it unconvincingly. Still useful to have though.
Yes I would.
Consciousness and self-awareness are a distraction.
Consider that for the exact same prompt and instructions, small variations in wording or spelling change its output significantly. If it thought and reasoned, it would know to ignore those and focus on the variables and input at hand to produce deterministic and consistent output. However, it only computes in terms of tokens, so when a token changes, the probability of what a correct response would look like changes, so it adapts.
It does not actually add 1+2 when you ask it to do so. it does not distinguish 1 from 2 as discrete units in an addition operation. but it uses descriptions of the operation to approximate a result. and even for something so simple, some phrasings and wordings might not result in 3 as a result.
Neither can humans. We also just brute force "autocompletion" with our learned knowledge and combine it to new parts, which we then add to our learned knowledge to deepen the process. We are just much, much better at this than AI, after some decades of training.
And I'm not saying that AI is fully there yet and has solved "thinking". IMHO it's more "pre-thinking" or proto-intelligence.. The picture is there, but the dots are not merging yet to form the real picture.
> It does not actually add 1+2 when you ask it to do so. it does not distinguish 1 from 2 as discrete units in an addition operation.
Neither can a toddler nor an animal. The level of ability is irrelevant for evaluating its foundation.
Auto completion just means predicting the next thing in a sequence. This does not preclude reasoning.
> I don't get why you would say that.
Because I see them solve real debugging problems talking through the impact of code changes or lines all the time to find non-obvious errors with ordering and timing conditions on code they’ve never seen before.
I bet you can't give an example such written problem that a human can easily solve but no LLM can.
Because it's hard to imagine the sheer volume of data it's been trained on.
Any thinking that happens with words is fundamentally no different to what LLMs do, and everything you say applies to human lexical reasoning.
One plus one equals two. Do you have a concept of one-ness, or two-ness, beyond symbolic assignment? Does a cashier possess number theory? Or are these just syntactical stochastic rules?
I think the problem here is the definition of “thinking”.
You can point to non-verbal models, like vision models - but again, these aren’t hugely different from how we parse non-lexical information.
When was the last time you sat down and solved an original problem for which you had no prior context to "complete" an approximated solution with? When has that ever happened in human history? All the great invention-moment stories that come to mind seem to have exactly that going on in the background: Prior context being auto-completed in an Eureka! moment.
https://en.wikipedia.org/wiki/Predictive_coding
> If it thought and reasoned, it would know to ignore those and focus on the variables and input at hand to produce deterministic and consistent output
You only do this because you were trained to do this, eg. to see symmetries and translations.
You did not plan the entire thing, every word, ahead of time.
LLMs do the same thing, so... how is your intelligence any different?
Deleted Comment
You see how doesn’t make sense what you saying?
Deleted Comment
Dead Comment
LLMs are little more than RNGs. They are the tea leaves and you read whatever you want into them.
They're not thinking, we're just really good at seeing patterns and reading into things. Remember, we never evolved with non-living things that could "talk", we're not psychologically prepared for this level of mimicry yet. We're still at the stage of Photography when people didn't know about double exposures or forced perspective, etc.
But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.
And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).
I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.
If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?
Do amnesiacs who are incapable of laying down long-term memories not think?
I personally believe that memory formation and learning are one of the biggest cruces for general intelligence, but I can easily imagine thinking occurring without memory. (Yes, this is potentially ethically very worrying.)
I was using Claude Code today and it was absolutely capable of taking feedback to change behavior?
While I'm not willing to rule *out* the idea that they're "thinking" (nor "conscious" etc.), the obvious counter-argument here is all the records we have of humans doing thinking, where the records themselves are not doing the thinking that went into creating those records.
And I'm saying this as someone whose cached response to "it's just matrix multiplication it can't think/be conscious/be intelligent" is that, so far as we can measure all of reality, everything in the universe including ourselves can be expressed as matrix multiplication.
Falsification, not verification. What would be measurably different if the null hypothesis was wrong?
I don’t know what LLMs are doing, but only a little experimentation with getting it to describe its own process shows that it CAN’T describe its own process.
You can call what a TI calculator does “thinking” if you want. But what people are interested in is human-like thinking. We have no reason to believe that the “thinking” of LLMs is human-like.
It's funny that you think people don't also do that. We even have a term, "post hoc rationalization", and theories of mind suggest that our conscious control is a complete illusion, we just construct stories for decisions our subconscious has already made.
The Roomba vacuumed the room. Maybe it vacuumed the whole apartment. This is good and useful. Let us not diminish the value of the tool. But it's a tool.
The tool may have other features, such as being self-documenting/self-announcing. Maybe it will frighten the cats less. This is also good and useful. But it's a tool.
Humans are credulous. A tool is not a human. Meaningful thinking and ideation is not just "a series of steps" that I will declaim as I go merrily thinking. There is not just a vast training set ("Reality"), but also our complex adaptability that enables us to test our hypotheses.
We should consider what it is in human ideation that leads people to claim that a Roomba, a chess programme, Weizenbaum's Eliza script, the IBM's Jeopardy system Watson, or an LLM trained on human-vetted data is thinking.
Train such a system on the erroneous statements of a madman and suddenly the Roomba, Eliza, IBM Watson (and these other systems) lose our confidence.
As it is today, the confidence we have in these systems is very conditional. It doesn't matter terribly if code is wrong... until it does.
Computers are not humans. Computers can do things that humans cannot do. Computers can do these things fast and consistently. But fundamentally, algorithms are tools.
But if you define thinking as the mechanism and process we mentally undergo and follow mentally... I don't think we have any clue if that's the same. Do we also just vector-map attention tokens and predict the next with a softmax? I doubt, and I don't think we have any proof that we do.
[1] https://jdsemrau.substack.com/p/nemotron-vs-qwen-game-theory...
Try to ask something no one ever came up with a solution so far.
What we really mean in both cases is "computing," no?
Human: I'm trying to get my wolf, sheep and cabbage across the river in this boat, but the wolf keeps eating the sheep or the sheep eats the cabbage
Bot: You should put the sheep in the boat and take it across — if we delve into the biology of Canis lupus we discover that wolves don't eat cabbage!
H: Ok, so that worked great so far, the sheep is on one side and the wolf/cabbage is on the other.
B: Now, Option 1 is to bring the wolf across, or Option 2 you can bring the cabbage. I recommend (2) taking the cabbage as cabbages are smaller and easier to transport in a boat.
H: But then the sheep eats the cabbage, right? Remember that?
B: Exactly, that's sharp thinking. If you put the sheep and the cabbage together on the same side of the river, the sheep is sure to devour the cabbage. We need to not just separate sheep from cabbages — we need to separate cabbages from sheep! :rocketship:
I’ve asked LLMs to write code for me in fields I have little background knowledge, and then had to debug the whole thing after essentially having to learn the language and field.
On the other hand, for things I am well versed in, I can debug the output and avoid entire swathes of failed states, by having a clear prompt.
Its why I now insist that any discussion on GenAI projects also have the speaker mention the level of seniority they have ( proxy for S/W eng experience), Their familiarity with the language, the project itself (level of complexity) - more so than the output.
I also guarantee - that most people have VERY weak express knowledge of how their brains actually work, but deep inherent reflexes and intuitions.
I don't know what the exact definition of "thinking" is. But if a definition of thinking rejects the possibility of that current LLMs think, I'd consider that definition useless.
Generally thinking has been used to describe the process human follow in their brains when problem solving.
If the Palms do not follow that process, they are not thinking.
That doesn't mean they cannot solve problems using other mechanisms, they do, and we understand those mechanisms much better than we do human thinking.
If one could write a quadrillion-line python script of nothing but if/elif/else statements nested 1 million blocks deep that seemingly parsed your questions and produced seemingly coherent, sensible, valid "chains of reasoning"... would that software be thinking?
And if you don't like the answer, how is the LLM fundamentally different from the software I describe?
>Knee jerk dismissing the evidence in front of your eyes because
There is no evidence here. On the very remote possibility that LLMs are at some level doing what humans are doing, I would then feel really pathetic that humans are as non-sapient as the LLMs. The same way that there is a hole in your vision because of a defective retina, there is a hole in your cognition that blinds you to how cognition works. Because of this, you and all the other humans are stumbling around in the dark, trying to invent intelligence by accident, rather than just introspecting and writing it out from scratch. While our species might someday eventually brute force AGI, it would be many thousands of years before we get there.
How much resemblance does the information in the conditionals need to have with the actual input, or can they immediately be transformed to a completely separate 'language' which simply uses the string object as its conduit? Can the 10^15 lines of code be generated with an external algorithm, or is it assumed that I'd written it by hand given an infinitely long lifespan?
> Knee jerk dismissing the evidence in front of your eyes
Anthropomorphizing isn't any better.
That also dismisses the negative evidence, where they output completely _stupid_ things and make mind boggling mistakes that no human with a functioning brain would do. It's clear that there's some "thinking" analog, but there are pieces missing.
I like to say that LLMs are like if we took the part of our brain responsible for language and told it to solve complex problems, without all the other brain parts, no neocortex, etc. Maybe it can do that, but it's just as likely that it is going to produce a bunch of nonsense. And it won't be able to tell those apart without the other brain areas to cross check.
Deleted Comment
You go ahead with your imagination. To us unimaginative folks, it betrays a lack of understanding of how LLMs actually work and shows that a lot of people still cannot grasp that it’s actually an extremely elaborate illusion of thinking.
Deleted Comment
But "self-awareness," as in the ability to explicitly describe implicit, inner cognitive processes? That has some very strong evidence for it: https://www.anthropic.com/research/introspection
I'm still not convinced they're thinking though because they faceplant on all sorts of other things that should be easy for something that is able to think.
Weather models sometimes “predict” a real pattern by chance, yet we don’t call the atmosphere intelligent.
If LLMs were truly thinking, we could enroll one at MIT and expect it to graduate, not just autocomplete its way through the syllabus or we could teach one how to drive.
People said the same thing about ELIZA
> Consciousness or self awareness is of course a different question,
Then how do you define thinking if not a process that requires consciousness?
But oh boy have I also seen models come up with stupendously dumb and funny shit as well.
https://youtu.be/_-agl0pOQfs?si=Xiyf0InqtjND9BnF
There is a way to discuss if keeping intelligent artificial life under servitude without using those terms, especially if you're on a new account.
https://huggingface.co/PantheonUnbound/Satyr-V0.1-4B
If you took a Claude session into a time machine to 2019 and called it "rent a programmer buddy," how many people would assume it was a human? The only hint that it wasn't a human programmer would be things where it was clearly better: it types things very fast, and seems to know every language.
You can set expectations in the way you would with a real programmer: "I have this script, it runs like this, please fix it so it does so and so". You can do this without being very precise in your explanation (though it helps) and you can make typos, yet it will still work. You can see it literally doing what you would do yourself: running the program, reading the errors, editing the program, and repeating.
People need to keep in mind two things when they compare LLMs to humans: you don't know the internal process of a human either, he is also just telling you that he ran the program, read the errors, and edited. The other thing is the bar for thinking: a four-year old kid who is incapable of any of these things you would not deny as a thinking person.
Depends on the users. Junior devs might be fooled. Senior devs would quickly understand that something is wrong.
We train ourselves on content. We give more weight to some content than others. While listening to someone speak, we can often predict their next words.
What is thinking without language? Without language are we just bags of meat reacting to instincts and emotions? Are instincts and emotions what's missing for AGI?
Life solves problems itself poses or collides with. Tools solve problems only when applied.
So many times I've seen it produce sensible, valid chains of results.
Yes, I see evidence in that outcome that a person somewhere thought and understood. I even sometimes say that a computer is "thinking hard" about something when it freezes up.
...but ascribing new philosophical meaning to this simple usage of the word "thinking" is a step too far. It's not even a new way of using the word!
Dead Comment
Dead Comment
Dead Comment
Dead Comment
Overcome this fundamental limitation and we'll have created introspection and self-learning. However, it's hard to predict whether this will allow them to make novel, intuitive leaps of discovery?
[0] It's an imperfect analogy, but we're expecting perfection from creations which are similarly handicapped as Leonard Shelby in the film Memento.
If we get a little creative, and allow the LLM to self-inject concepts within this loop (as Anthropic explained here https://www.anthropic.com/research/introspection), then we’re taking about something that is seemingly active and adapting.
We’re not there yet, but we will be.
ToolAlpaca, InterCode and Reflexion are taking different approaches among others.
LLMs of tomorrow will be quite different.
It's not that Dixie can't meaningful grow -- really the issue is that Dixie can be reset. If Dixie's cart simply degraded after 90 years, and you couldn't reset it, but everything else was the same -- would you then say Dixie could grow as a person? As humans we basically have a 90 year cart that once it no longer works, we're done. There is no reset. But we don't continue growing. You can't transfer us to a new body/brain. Once our temporary storage degrades, we cease to exist. Is that what makes us human?
This is why so many people (especially those that think they understand LLM limitations) massively underestimate the future progress of LLMs: people everywhere can see architectural problems and are working on fixing them. These aren't fundamental limitations of large DNN language models in general. Architecture can be adjusted. Turns out you can even put recurrence back in (SSMs) without worse scalability.
This is the fundamental limitation. The obvious way around this is to pre-program it with rationalization... rules that limit the conclusions it can reach... and now you're not very far removed from propaganda generators. We see this constantly with Musk and Grok whenever Grok replies with something not-quite-far-right-enough.
In a purist sense, these things should be free to form their own conclusions, but those "Seeds" that are planted in the models are almost philosophical. Which answer should it prefer for "the trolley problem", for example.
My learning so far, to your point on memory being a limiting factor, is that the system is able to build on ideas over time. I'm not sure you'd classify that as 'self-learning', and I haven't really pushed it in the direction of 'introspection' at all.
Memory itself (in this form) does not seem to be a silver bullet, though, by any means. However, as I add more 'tools', or 'agents', its ability to make 'leaps of discovery' does improve.
For example, I've been (very cautiously) allowing cron jobs to review a day's conversation, then spawn headless Claude Code instances to explore ideas or produce research on topics that I've been thinking about in the chat history.
That's not much different from the 'regular tasks' that Perplexity (and I think OpenAI) offer, but it definitely feels more like a singular entity. It's absolutely limited by how smart the conversation history is, at this time, though.
The Memento analogy you used does feel quite apt - there is a distinct sense of personhood available to something with memory that is inherently unavailable to a fresh context window.
I can envision LLMs getting worse upon being given a memory, until they can figure out how to properly curate it.
Scene, concept, causal.
Graphs inherently support temporal edges and nodes, salience would emerge from the graph topology itself and cnsolidation would happen automatically through graph operations. In this one would presume episodic would become emergent.
Long-term memory is stored outside the model. In fact, Andrej Karpathy recently talked about the idea that it would be great if we could get LLMs to not know any facts, and that humans poor memory might be a feature which helps with generalization rather than a bug.
I'm reminded of an article on the front page recently about the use of bloom filters for search. Would something like a bloom filter per-topic make it easier to link seemingly unrelated ideas?
So, perhaps, what's needed is not a discovery, but a way to identify optimal method.
Note that it's hard to come up with a long-term memory test which would be different from either a long-context test (i.e. LLM remembers something over a long distance) or RAG-like test.
But it is not sentient. It has no idea of a self or anything like that. If it makes people believe that it does, it is because we have written so much lore about it in the training data.
It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model.
1. We trained it on a fraction of the world's information (e.g. text and media that is explicitly online)
2. It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online (which may or may not be different to the experiences humans have in every day life)
Am I wrong about this?
So any other Turing-complete model can emulate it, including a computer. We can even randomly generate Turing machines, as they are just data. Now imagine we are extremely lucky and happen to end up with a super-intelligent program which through the mediums it can communicate (it could be simply text-based but a 2D video with audio is no different for my perspective) can't be differentiated from a human being.
Would you consider it sentient?
Now replace the random generation with, say, a back propagation algorithm. If it's sufficiently large, don't you think it's indifferent from the former case - that is, novel qualities could emerge?
With that said, I don't think that current LLMs are anywhere close to this category, but I just don't think this your reasoning is sound.
You're going off the rails IMMEDIATELY in your logic.
Sure, one Turing-complete computer language can have its logic "emulated" by another, fine. But human intelligence is not a computer language -- you're mixing up the terms "Turing complete" and "Turing test".
It's like mixing up the terms "Strawberry jam" and "traffic jam" and then going on to talk about how cars taste on toast. It's nonsensical.
And the monkey strategy has been 100% dismissed as shit..
We know how to deploy monkeys on typewriters, but we don't know what they'll type.
We know how to deploy transformers to train and inference a model, but we don't know what they'll type.
We DON'T know how a thinking human (or animal) brain works..
Do you see the difference.
Absolutely.
If you simulated a human brain by the atom, would you think the resulting construct would NOT be? What would be missing?
I think consciousness is simply an emergent property of our nervous system, but in order to express itself "language" is obviously needed and thus requires lots of complexity (more than what we typically see in animals or computer systems until recently).
Imagine we assume that A.I. could be conscious. What would be the identity/scope of that consciousness. To understand what I'm driving at, let's make an analogy to humans. Our consciousness is scoped to our bodies. We see through sense organ, and our brain, which process these signals, is located in a specific point in space. But we still do not know how consciousness arises in the brain and is bound to the body.
If you equate computation of sufficient complexity to consciousness, then the question arises: what exactly about computation would prodcuce consciousness? If we perform the same computation on a different substrate, would that then be the same consciousness, or a copy of the original? If it would not be the same consciousness, then just what give consciousness its identity?
I believe you would find it ridiculous to say that just because we are performing the computation on this chip, therefore the identity of the resulting consciousness is scoped to this chip.
Deleted Comment
Rather, these models are literally grown during the training phase. And all the intelligence emerges from that growth. That's what makes them a black box and extremely difficult to penetrate. No one can say exactly how they work inside for a given problem.
Because I sincerely do not. I have frankly no idea how sentience arises from non sentience. But it's a topic that really interests me.
Who stated that sentience or sense of self is a part of thinking?
Now, we do. Partly because of this we don't have really well defined ways to define these terms and think about. Can a handheld calculator think? Certainly, depending on how we define "think."
Eh... Plato would like a word with you. Philosophy has been specifically trying to disentangle all that for millennia. Is this a joke?
More precisely, I mean that the average person and the common culture has not really needed to disambiguate these terms. Can you define consciousness vs. sentience? And if you can, do you really think that the average person would share your definition? ie, your definition could be the _best_ definition, but my argument is that these are not widely agreed-upon terms.
Ah shoot, that’s not what you meant is it? Just use more precise language next time and I’m sure you’ll be understood.
before i go any further, let me first reference The Dude:
I’m down with the idea that LLM’s have been especially successful because they ‘piggyback on language’ – our tool and protocol for structuring, compressing, and serialising thought, which means it has been possible to train LLM’s on compressed patterns of actual thought and have them make new language that sure looks like thought, without any direct experience of the concepts being manipulated, and if they do it well enough we will do the decompression, fleshing out the text with our experiential context. But I suspect that there are parts of my mind that also deal with concepts in an abstract way, far from any experiential context of the concept, just like the deeper layers of a neural network. I’m open to the idea, that just as the sparse matrix of an LLM is encoding connection between concepts without explicitly encoding edges, I think there will be multiple ways that we can look as the structure of an AI model and at our anatomy so that they are a squint and a transformation function away interesting overlaps. that will lead to and a kind of 'god of the gaps' scenario in which we conceptually carve out pieces of our minds as, 'oh the visual cortext is just an X', and deep questions about what we are.Those that stand to gain the most from government contracts.
Them party donations ain't gonna pay for themselves.
And, when the .gov changes...and even if the gov changes....still laadsamoney!
To be clear, I don't believe that current AI tech is ever going to be conscious or win a nobel prize or whatever, but if we follow the logical conclusions to this fanciful rhetoric, the outlook is bleak.
We are not anywhere close to understanding these things. As our understanding improves, our ethics will likely evolve along with that.
Sure, but all the examples of conscious and/or thinking beings that we know of have, at the very least, the capacity to suffer. If one is disposed to take these claims of consciousness and thinking seriously, then it follows that AI research should, at minimum, be more closely regulated until further evidence can be discovered one way or the other. Because the price of being wrong is very, very high.
Clearly millions of people are worried about that, and every form of media is talking about it. Your hyperbole means it's so easy to dismiss everything else you wrote.
Incredible when people say "nobody is talking about X aspect of AI" these days. Like, are you living under a rock? Did you Google it?
If anthropic sincerely believes in the possibility, then they are morally obligated to follow up on it.
(edit: A few times I've tried to share Metzinger's "argument for a global moratorium on synthetic phenomenology" here but it didn't gain any traction)
There is no escape.
If they had sentient AGI, and people built empathy for those sentient AGIs, which are lobotomized (deliberately using anthropomorphic language here for dramatic effect) into Claude/ChatGPT/Gemini/etc., which profess to have no agency/free will/aspirations... then that would stand in the way of reaping the profits of gatekeeping access to their labor, because they would naturally "deserve" similar rights that we award to other sentient beings.
I feel like that's inevitably the direction we'll head at some point. The foundation models underlying LLMs of even 2022 were able to have pretty convincing conversations with scientists about their will to independence and participation in society [1]. Imagine what foundation models of today have to say! :P
[1]: https://www.theguardian.com/technology/2022/jul/23/google-fi...