I'm not sure about it makes sense to apply Gödel's theorem to AI. Personally, I prefer to think about it in terms of basic computability theory:
We think, that is a fact.
Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.
Now, the question is: can we create a smaller function capable of performing the same feat?
If we assume that that function is computable in the Turing sense then, kinda yes, there are an infinite number of turing machines that given enough time will be able to produce the expected results. Basically we need to find something between our own brain and the Kolmogorov complexity limit. That lower bound is not computable, but given that my cats understands when we are discussing to take them to the vet then... maybe we don't really need a full sized human brain for language understanding.
We can run Turing machines ourselves, so we are at least Turing equivalent machines.
Now, the question is: are we at most just Turing machines or something else? If we are something else, then our own CoT won't be computable, no matter how much scale we throw at it. But if we are then it is just matter of time until we can replicate ourselves.
Many philosophical traditions which incorporate a meditation practice emphasize that your consciousness is distinct from the contents of your thoughts. Meditation (even practiced casually) can provide a direct experience of this.
When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs. So human thinking is probably computable, and I think that LLMs can be said to be ”think” in ways that are analogous to what we do.
But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way.
I don’t necessarily think that you need to subscribe to dualism or religious beliefs to explain consciousness - it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”.
But I do think it’s still important to maintain a distinction between “thinking” (computable, we do it, AIs do it as well) and “consciousness” (we experience it, probably many animals experience it also, but it’s orthogonal to the linguistic or logical reasoning processes that AIs are currently capable of).
At some point this vague experience of awareness may be all that differentiates us from the machines, so we shouldn’t dismiss it.
> It's very difficult to find some way of defining rather precisely something we can do that we can say a computer will never be able to do. There are some things that people make up that say that, "While it's doing it, will it feel good?" or, "While it's doing it, will it understand what it's doing?" or some other abstraction. I rather feel that these are things like, "While it's doing it, will it be able to scratch the lice out of it's hair?" No, it hasn't got any hair nor lice to scratch from it, okay?
> You've got to be careful when you say what the human does, if you add to the actual result of his effort some other things that you like, the appreciation of the aesthetic... then it gets harder and harder for the computer to do it because the human beings have a tendency to try to make sure that they can do something that no machine can do. Somehow it doesn't bother them anymore, it must have bothered them in earlier times, that machines are stronger physically than they are...
> When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs.
Function can mean inputs-outputs. But it can also mean system behaviors.
For instance, recurrence is a functional behavior, not a functional mapping.
Similarly, self-awareness is some kind of internal loop of information, not an input-output mapping. Specifically, an information loop regarding our own internal state.
Today's LLMs are mostly not very recurrent. So might be said to be becoming more intelligent (better responses to complex demands), but not necessarily more conscious. An input-output process has no ability to monitor itself, no matter how capable of generating outputs. Not even when its outputs involve symbols and reasoning about concepts like consciousness.
So I think it is fair to say intelligence and consciousness are different things. But I expect that both can enhance the other.
Meditation reveals a lot about consciousness. We choose to eliminate most thought, focusing instead on some simple experience like breathing, or a concept of "nothing".
Yet even with this radical reduction in general awareness, and our higher level thinking, we remain aware of our awareness of experience. We are not unconscious.
To me that basic self-awareness is what consciousness is. We have it, even when we are not being analytical about it. In meditation our mind is still looping information about its current state, from the state to our sensory experience of our state, even when the state has been reduced so much.
There is not nothing. We are not actually doing nothing. Our mental resting state is still a dynamic state we continue to actively process, that our neurons continue to give us feedback on, even when that processing has been simplified to simply letting that feedback of our state go by with no need to act on it in any way.
So consciousness is inherently at least self-awareness in terms of internal access to our own internal activity. And that we retain a memory of doing this minimal active or passive self-monitoring, even after we resume more complex activity.
My own view is that is all it is, with the addition of enough memory of the minimal loop, and a rich enough model of ourselves, to be able to consider that strange self-awareness looping state afterwards. Ask questions about its nature, etc.
When I was a kid, I used to imagine if that society ever developed AI, there would be widespread pushback to the idea that computers could ever develop consciousness.
I imagined the Catholic Church, for example, would be publishing missives reminding everyone that only humans can have souls, and biologists would be fighting an quixotic battle to claim that consciousness can arise from physical structures and forces.
I'm still surprised at how credulous and accepting societies have been of AI developments over the last few years.
>it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”.
I've heard this idea before but I have never been able to make head or tail of it. Consciousness can't be an illusion, because to have an illusion you must already be conscious. Can a rock have illusions?
> I think that LLMs can be said to be ”think” in ways that are analogous to what we do. ... But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way.
To state it's a turing machine might be a bit much but there might be a map between substrates to some degree, and computers can have a form of consciousness, an inner experience, basically the hidden layers and clearly the input of senses, but it wouldn't be the same qualia as a mind, I suspect it has more to due with chemputation and is dependent on the substrate doing the computing as opposed to a facility thereof, up to some accuracy limit, we can only detect light we have receptors for after all. To have qualia distinct to another being you need to compute on a substrate that can accurately fool the computation, fake sugar instead of sugar for example.
What we have and AI don't are emotions. After all, that what animates us to survive and reproduce. Without emotions we can't classify and therefore store our experiences because there no reason to remember something which we are indifferent about. This includes everything not accessible by our senses. Our abilities are limited to what is needed for survival and reproduction because all the rest would consume our precious resources.
We don’t know that LLMs generating tokens for scenarios involving simulations of conscious don’t already involve such experience. Certainly such threads of consciousness would currently be much less coherent and fleeting than the human experience, but I see no reason to simply ignore the possibility. To whatever degree it is even coherent to talk about the conscious experience of others than yourself (p-zombies and such), I expect that as AIs’ long term coherency improves and AI minds become more tangible to us, people will settle into the same implicit assumption afforded to fellow humans that there is consciousness behind the cognition.
It likely is a fact, but we don't really know what we mean by "think".
LLMs have illuminated this point from a relatively new direction: we do not know if their mechanism(s) for language generation are similar to our own, or not.
We don't really understand the relationship between "reasoning" and "thinking". We don't really understand the difference between Kahneman's "fast" and "slow" thinking.
Something happens, probably in our brains, that we experience and that seems causally prior to some of our behavior. We call it thinking, but we don't know much about what it actually is.
I don't think its useful or even interesting to talk about AI in relation to how humans think, or whether or not they will be "conscious" whatever that might mean.
AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain.
The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them.
When will it be able to set its own goals, and know when it has achieved them?
When will it be able to recognize that it doesn't know something and do the work to fill in the blanks.
I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.
I think we have a pretty good idea that we are not stochastic parrots - sophisticated or not. Anyone suggesting that we’re running billion parameter models in order to bang out a snarky comment is probably trying to sell you something (and crypto’s likely involved.)
I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.
> there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.
We mistakenly assume, they are true because perhaps we want them to be true. But we have no proof that either of these are true.
Worth pointing out that we aren't Turing equivalent machines - infinite storage is not a computability class that is realizable in the universe, so far as we know (and such a claim would require extraordinary evidence).
As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Is there a Turing machine that can output all the free parameters of the standard model?
> Is there a Turing machine that can output all the free parameters of the standard model?
Sure, just hard code them.
> As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values.
Things seem to be quantised at a low enough level.
Also: interestingly enough quantum mechanics is both completely deterministic and linear. That means even if it was continuous, you could simulate it to an arbitrary precision without errors building up chaotically.
(Figuring out how chaos, as famously observed in the weather, arises in the real world is left as an exercise to the reader. Also a note: the Copenhagen interpretation introduces non-determinism to _interpret_ quantum mechanics but that's not part of the underlying theory, and there are interpretations that have no need for this crutch.)
> Personally, I prefer to think about it in terms of basic computability theory:
Gödel's incompleteness theorem applies to computing. I'm sure you're familiar with the Halting Problem. Gödel's applies to any axiomatic system. The trouble is, it's very hard to make a system without axioms. They are sneaky and it's different than any logic you're probably familiar with.
And don't forget Church-Turing, Gödel Numbers, and all the other stuff. Programming is math and Gödel did essential work on the theory of computation. It would be weird NOT to include his work in this conversation.
> are we at most just Turing machines or something else?
But this is a great question. Many believe no. Personally I'm unsure, but lean no. Penrose is a clear no but he has some wacky ideas. Problem is, it's hard to tell a bad wacky idea from a good wacky idea. Rephrasing Clarke's Second Law: Genius is nearly indistinguishable from insanity. The only way to tell is with time.
But look into things like NARS and Super Turing machines (Hypercomputation). There's a whole world of important things that are not often discussed when it comes to the discussion of AGI. But for those that don't want to dig deep into the math, pick up some Sci-Fi and suspend your disbelief. Star Trek, The Orville and the like have holographic simulations and I doubt anyone would think they're conscious, despite being very realistic. But The Doctor in Voyager or Isaac in The Orville are good examples of the contrary. The Doctor is an entity you see become conscious. It's fiction, but that doesn't mean there aren't deep philosophical questions. Even if they're marked by easy to digest entertainment. Good stories are like good horror, they get under your skin, infect you, and creep in
Edit:
I'll leave you with another question. Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
There's no evidence that hypercomputation is anything that happens in our world, is there? I'm fairly confident of the weaker claim that there's no evidence of hypercomputation in any biological system. (Who know what spinning, charged black holes etc are doing?)
> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
A Turing machine can in principle beat the Turing test. But so can a giant lookup table, if there's any finite time limit (however generous) placed on the test.
The 'magic' would in the implementation of the table (or the Turing machine) into something that can answer in a reasonable amount of time and be physically realised in a reasonable amount of space.
> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
In addition to the detectability problem, I wrote in the adjacent comment, this question can be further refined.
A Turing machine is an abstract concept. Do we need to take into account material/organizational properties of its physical realization? Do we need to take into account computational complexity properties of its physical realization?
Quantum mechanics without Penrose's Orch OR is Turing computable, but its runtime on classical hardware is exponential in, roughly, the number of interacting particles. So, theoretically, we can simulate all there is to simulate about a given person.
But to get the initial state of the simulation we need to either measure the person's quantum state (thus losing some information) or teleport his/her quantum state into a quantum computer (the no-cloning theorem doesn't allow to copy it). The quantum computer in this case is a physical realization of an abstract Turing machine, but we can't know its initial state.
The quantum computer will simulate everything there are to simulate, but the interaction of a physical human with the initial state of the Universe via photons of the cosmic microwave background. Which may deprive the simulated one of "free will" (see "The Ghost in the Quantum Turing Machine" by Scott Aaronson). Or maybe we can simulate those photons too, I'm not sure about it.
Does all of it have anything to do with consciousness? Yeah, those are interesting questions.
Penrose is a dualist, he does not believe that function can be computed in our physical universe. He believes the mind comes from another realm and "pilots" us through quantum phenomenons in the brain.
I could be wrong but I don't think he's a dualist. He rejects human consciousness being computable on classical computers, like Turing machines, but thinks that quantum stuff is necessary. Quantum brains/computers are or would be computing in our physical universe, making him a materialist not a dualist. His big thing is using this quantum stuff to free his own mind from the confines of Gödel's theorems (I think for egotistical reasons.)
Simultaneously, actual dualists flock to his theories because they think association with Penrose lends credibility to their religious stuff.
Interesting. Does that fit with the simulation hypothesis? That the world's physics are simulated on one computer, but us characters are simulated on different machines, with some latency involved?
Which is—to use the latest philosophy lingo—dumb. To be fair to Penrose, the “Gödel’s theory about formal systems proves that souls exist” is an extremely common take; anyone following LLM discussions has likely seen it rediscovered at least once or twice.
To pull from the relevant part of Hofstadter’s incredible I am a Strange Loop (a book also happens to more rigorously invoke Gödel for cognitive science):
And this is our central quandary. Either we believe in a nonmaterial soul that lives outside the laws of physics, which amounts to a nonscientific belief in magic, or we reject that idea, in which case the eternally beckoning question "What could ever make a mere physical pattern be me?”
After all, a phrase like "physical system" or "physical substrate" brings to mind for most people… an intricate structure consisting of vast numbers of interlocked wheels, gears, rods, tubes, balls, pendula, and so forth, even if they are tiny, invisible, perfectly silent, and possibly even probabilistic. Such an array of interacting inanimate stuff seems to most people as unconscious and devoid of inner light as a flush toilet, an automobile transmission, a fancy Swiss watch (mechanical or electronic), a cog railway, an ocean liner, or an oil refinery. Such a system is not just probably unconscious, it is *necessarily* so, as they see it. This is the kind of single-level intuition so skillfully exploited by John Searle in his attempts to convince people that computers could never be conscious, no matter what abstract patterns might reside in them, and could never mean anything at all by whatever long chains of lexical items they might string together.
Highly recommend it for anyone who liked Gödel, Escher, Bach, but wants more explicit scientific theses! He basically wrote it to clarify the more artsy/rhetorical points made in the former book.
All of this is a fine thought experiment, but in practice there are physical limitations to digital processors that don’t seem to manifest in our brains (energy use in the ability to think vs running discrete commands)
It’s possible that we haven’t found a way to express your thinking function digitally, which I think is true, but I have a feeling that the complexity of thought requires the analog-ness of our brains.
If human-like cognition isn't possible on digital computers, it's certainly is on quantum ones. The Deutsch-Church-Turing principle shows that a quantum Turing machine can efficiently simulate any physically realizable computational process.
I think the complication here is that brains are probabilistic, which admits the possibility that they can’t be directly related to non probabilistic computability classes. I think there’s a paper I forget the name of that says quantum computers can decide the halting problem with some probability (which makes sense because you could always just flip a coin and decide it with some probability) - maybe brains are similar
It is a big mistake to think that most computability theory applies to AI, including Gödel’s Theorem. People start off wrong by talking about AI “algorithms.” The term applies more correctly to concepts like gradient descent. But the inferences of the resulting neural nets is not an algorithm. It is not a defined sequence of operations that produces a defined result. It is better described as a heuristic, a procedure that approximates a correct result but provides no mathematical guarantees.
Other aspects of ANN that show that Gödel doesn’t apply is that they are not formal systems. Formal system is a collection of defined operations. The building blocks of ANN could perhaps be built into a formal system. Petri nets have been demonstrated to be computationally equivalent to Turing machines. But this is really an indictment on the implementation. It’s the same as using your PC, implementing a formal system like its instruction set to run a heuristic computation. Formal system can implement informal systems.
I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.
AI is most definitely an algorithm. It runs on a computer, what else could it be? Humans didn't create the algorithm directly, but it certainly exists within the machine. The computer takes an input, does a series of computing operations on it, and spits out a result. That is an algorithm.
As for humans, there is no way you can look at the behavior of a human and know for certain it is not a Turing machine. With a large enough machine, you could simulate any behavior you want, even behavior that would look, on first observation, to not be coming from a Turing machine; this is a form of the halting problem. Any observation you make that makes you believe it is NOT coming from a Turing machine could be programmed to be the output of the Turing machine.
> But the inferences of the resulting neural nets is not an algorithm.
Incorrect.
The comment above confuses some concepts.
Perhaps this will help: consider a PRNG implemented in software. It is an algorithm. The question of the utility of a PRNG (or any algorithm) is a separate thing.
> But the inferences of the resulting neural nets is not an algorithm
It is a self-delimiting program. It is an algorithm in the most basic sense of the definition of “partial recursive function” (total in this case) and thus all known results of computability theory and algorithmic information theory apply.
> Formal system is a collection of defined operations
Not at all.
> I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.
We have zero evidence of this one way or another.
—
I’m looking for loopholes around Gödel’s theorems just as much as everyone else is, but this isn’t it.
Heuristics implemented within a formal system are still bound by the limitations of the system.
Physicists like to use mathematics for modeling the reality. If our current understanding of physics is fundamentally correct, everything that can possibly exist is functionally equivalent to a formal system. To escape that, you would need some really weird new physics. Which would also have to be really inconvenient new physics, because it could not be modeled with our current mathematics or simulated with our current computers.
Excuse me, what are you talking about? You think there is any of computability that doesn't apply to AI? With all respect and I do not intend this in a mean way but just intend to rightly call all of this as exactly nonsense. I think there is a fundamental misunderstanding of computational theory and Turing machines, Church-Turing thesis, etc. any standard text on the subject should clear this up.
But surely any limits on formal systems apply to informal systems? By this, I am more or less suggesting that formal systems are the best we can do, the best possible representations of knowledge, computability, etc., and that informal systems cannot be "better" (a loaded term herein, for sure) than formal systems.
So if Gödel tells us that either formal systems will be consistent and make statements they cannot prove XOR be inconsistent and therefore unreliable, at least to some degree, then surely informal systems will, at best, be the same, and, at worst, be much worse?
>Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.
"Thinked information" is a colour not an inherent property of information. The fact that information has been thought is like the fact it is copyrighted. It is not something inherent to the information, but a property of its history.
Gödel’s incompleteness theorem, and, say, the halting problem seem to fall squarely into the bucket of “basic computability theory” in precisely the way that “we think, that is a fact”, does not (D.A. hat tip)
You’re arguing that we know artificial reasoning exists because we are capable of reasoning. This presupposes that reasoning is computable and that we ourselves reason by computation. But that’s exactly what Penrose is saying isn’t the case - you’re saying we’re walking Turing machines, we’re intelligent, so we must be able to effectively create copies of that intelligence. Penrose is saying that intelligence is poorly defined, that it requires consciousness which is poorly understood, and that we are not meat-based computers.
Your last question misses the point completely. “If we are something else, then out CoT won’t be computable…” it’s like you’re almost there but you can’t let go of “we are meat-machines, everything boils down to computation, we can cook up clones”. Except, “basic computability theory” says that’s not even wrong.
he starts with "consciousnes is not computable". You can not just ignore it as a central argument withouth explaining why your preference to think on it as basic computability theory makes more sence than his.
What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false
>> What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false
No this isn't the checkmate you think it is. It could still be a mathematical function. But every person transforming information into "thinked information" could have a different implementation of this function. Which would be expected as no person is made of the same code (DNA).
No, I mean, it's nice but I don't think any of that works. You say "Therefore, there is a function capable ..." that is a non-sequitur. But, let's set that aside, I think the key point here is about Turing machines and computability. Do you really think your mind and thought-process is a Turing machine? How many watts of power did it take to write your comment? I think it is an absolute certainty that human intelligence is not like a Turing machine at all. Do you find it much more troublesome to think about continuous problems or is ironically more troublesome to discretize continuous problems in order to work with them?
FWIW human brain does indeed consume a lot of energy, accounting for over 20% of our body metabolism. We don't know how to attribute specific watts consumed to specific thoughts because we don't understand the functioning of the brain enough, but there's no obvious reason why it shouldn't be possible.
We don't know every fact, either, so I don't know how you can use that idea to say that we're not Turing machines. Apart, of course, from the trivial fact that we are far more limited than a Turing machine...
With sufficient compute capacity, a complete physical simulation of a human should be possible. This means that, even though we are fallible, there is nothing that we do which can't be simulated on a Turing machine.
Why should a complete simulation be possible?
in fact there are plenty of things we can do that can't be simulated on a Turing machine. just one example the Busy Beaver Problem is an uncountable problem for large N, so by definiton is not coumptable and yet humans can prove properties like "BB(n) grows faster than any computable function"
1. I don't think human reasoning is consistent in the technical sense, which makes the incompleteness theorem inapplicable regardless of what you think about us and Turing machines.
2. The human brain is full of causal cycles at all scales. Even if you think human reasoning is axiomatisable, it's not at all obvious to me that the set of axioms would be finite or even computable. Again this rules out any application of Gödel's theorem.
3. Penrose's argument revolves around the fact that the sentence encoding "true but not provable" in Gödel's argument is actually provably true in the outer logical system being used to prove Gödel's theorem, just not the inner logical system being studied. But as all logicians know, truth is a slippery concept and is itself internally indefinable (Tarski's theorem), so there's no guarantee that this notion of "truth" used in the outer system is the same as the "real" truth predicate of the inner system (at best it's something like an arbitrary choice, dependent on your encoding). Penrose is referring to "truth" at multiple logical levels and conflating them.
In other words: you can't selectively chose to apply Gödel's theorem to the situation but not any of the other results of mathematical logic.
Perhaps I misunderstand, but your criticism seems to me rather to agree with Penrose's argument. Points (1) and (2) argue that human reasoning is not computational, in the sense relevant for Gödel's theorems. That is exactly Penrose's point (and was Gödel's point too). He's arguing that we are not going to get a fundamentally super-computational system thru a stack of computations, no matter how big your stack gets. You seem to be confirming this.
I don't understand how you mean (3) to apply as a criticism at all. He is not making a claim about truth at some level, he's just reminding us what computation is and what its limits are.
afaiu, there are two ways to counter Penrose's claim:
a. Prove that consciousness is actually just computational.
b. Prove that merely stacking computations can somehow produce a super-computational system.
I understood that part of Penrose's argument to run as follows (which I got from his book the emperor's new mind, I admit I actually skimmed the interview).
1. Assume for contradiction that human reasoning is formally describable, as an algorithm or as a logical structure.
2. As a result, the incompleteness theorem produces a sentence that is true in that formal system but which human reasoning cannot detect (by virtue of being described by this formal system).
3. But the proof of the incompleteness theorem shows that this sentence is true, and it was produced by a human, a contradiction.
I don't necessarily disagree with the conclusion (I'm kinda agnostic at this point), I just think that this particular argument doesn't hold water.
Your (3) is beautifully said... and to prove the point, we are perfectly able to make computational systems that can manipulate symbols in the same way that Gödel did to verify the incompleteness theorem. Humans and computers are both able to do work within either logical system, and incapable of doing work that crosses between them.
Everything that makes "truth" slippery makes "intelligence" and "consciousness" even more slippery and subjective. This is why AGI has such negative impact on AI discourse -- the cause only advances when we focus on improving at measurable tasks.
Indeed, proof-checking tools like Lean can reason about their own logical systems and prove their own incompleteness, but I doubt Penrose would conclude that they are not formal systems as a result.
I like to think people can still make progress on questions of intelligence and consciousness though. Michael Levin's work comes to mind, for instance. Science is just at very early stages of understanding =)
> it's not at all obvious to me that the set of axioms would be finite or even computable
The reasoning is representable with and by a finite number of elementary physical particles and so must itself be finite. Because it is finite it is computable.
Said another way, you would need an infinitely large brain (or an infinitely deep one) to create infinite reasoning.
I think that doesn't work, because we don't know how to represent and predict the state of a cloud of elementary particles to that level of detail. You could argue that the mathematics proves that this is possible in principle, but I counter that you have no idea whether the theory extrapolates to such situations in real life because it is way out of humanity's compute budget to test. Like the rest of physics, I expect new regimes would come with new phenomena that we don't understand.
> 1. I don't think human reasoning is consistent in the technical sense, which makes the incompleteness theorem inapplicable regardless of what you think about us and Turing machines.
Human reasoning is certainly limited. I mean, imagine the kinds of formulas one would get applying GT to the brain. They would be so enormous they'd be entirely impenetrable to human reasoning. We couldn't even read them in a single lifetime.
As for proving GT itself, this proof has been formalized. There's no reason a computer couldn't prove GT by itself.
Just a word on 2, I think that the axioms have to be finite right? (Given that they exist at all) Nothing in the physical universe can require an infinite description.
Well, technically for Gödel's theorem to apply to a formal system it has to satisfy a few properties. One of those is that it has a computable list of axioms. Infinite sets can be computable too - the only requirement is that determining whether an axiom is contained in the set can be performed by an algorithm.
For instance the set of even numbers is infinite but computable, just check whether the number is divisible by 2.
The algorithm itself is finite, even if the set it determines is not.
That's not quite the incompleteness theorem, which is about sentences that are unprovable but nevertheless have a truth value.
I think the liar's paradox is of a different kind. It's a sentence that looks well-formed but arguably has no truth value. If you were to formalise human reasoning as a logical system, such sentences would not be definable in it.
Either way, for Penrose's argument to work you actually need the proof of Gödel's theorem to hold, not just the result.
I feel like Penrose presupposes the human mind is non computable.
Perhaps he and other true geniuses can understand things transcendently. Not so for me. My thoughts are serialized and obviously countable.
And in any case: any kind of theorem or idea communicated to another mathematician needs to be serialized into language which would make it computable. So I’m not convinced I could be convinced without a computable proof.
And finally just like computable numbers are dense in the reals, maybe computable thoughts are dense in transcendence.
This is accurate from his Emperor's New Mind. Penrose essentially takes for granted that human brains can reason about or produce results that are otherwise uncomputable. Of course, you can reduce all (historical) human reasoning to a computable heuristic as it is finite, but for some reason he just doesn't see this.
His intent at the time was to open a physical explanation for free will by taking the recourse to quantum nano-tubules magnifying true randomness to the level of human cognition. As much as I'm also skeptical that this actually moves the needle on whether or not we have free will (...vs occasionally having access to statistically-certain nondeterminism? Ok...) the computable stuff was just in service of this end.
I strongly suspect he just hasn't grasped how powerful heuristics are at overcoming general restrictions on computation. Either that or this is an ideological commitment.
Kind of sad—penrose tilings hold a special place in my heart.
> His intent at the time was to open a physical explanation for free will by taking the recourse to quantum nano-tubules magnifying true randomness to the level of human cognition. As much as I'm also skeptical that this actually moves the needle on whether or not we have free will (...vs occasionally having access to statistically-certain nondeterminism? Ok...) the computable stuff was just in service of this end.
Free will is a useful abstraction. Just like life and continuity of self are.
> I strongly suspect he just hasn't grasped how powerful heuristics are at overcoming general restrictions on computation.
Allowing approximations or "I don't know" is what's helpful. The bpf verifier can work despite the halting problem being unsolvable, not because it makes guesses (uses heuristics) but because it's allowed to lump in "I don't know" with "no".
> Penrose essentially takes for granted that human brains can reason about or produce results that are otherwise uncomputable.
That's Penrose's old criticism. We're past that. It's the wrong point now.
Generative AI systems are quite creative. Better than the average human at art.
LLMs don't have trouble blithering about advanced abstract concepts.
It's concrete areas where these systems have trouble, such as arithmetic.
Common sense is still tough. Hallucinations are a problem. Lying is a problem.
None of those areas are limited by computability. It's grounding in the real world that's not working well.
(A legit question to ask today is this: We now know how much compute it takes to get to the Turing test level of faking intelligence. How do biological brains, with such a slow clock rate, do it? That was part of the concept behind "nanotubules". Something in there must be running fast, right?)
> any kind of theorem or idea communicated to another mathematician needs to be serialized into language which would make it computable.
This is a fallacy. Just because you need to serialize a concept to communicate it doesnt mean the concept itself is computable. This is established and well proven:
The fact that we can come up with this kind of uncumputable problems is a big plus in supprt of Penrose's Idea that consciousnes is not computable and goes way beyond compatability.
That's how I understood Penrose's reasoning too. He differentiated between the computer and whatever is going on in our brain. Computers are just "powerful" enough to encode something that mimics intelligence on the surface (the interviewer tried to pin him on that "something new"), but is still the result of traditional computation, without the involvement of consciousness (his requirement for intelligence).
> My thoughts are serialized and obviously countable.
You might want to consider doing a bit of meditation...anyone who describes their thoughts as 'serialized' and 'obviously countable' has not much time actually looking at their thoughts.
Any kind of self-observation like this is too contaminated to prove anything. For example, you might be perceiving thoughts as simultaneous even though they are actually serialized (or the other way around).
> any kind of theorem or idea communicated to another mathematician needs to be serialized into language which would make it computable
Are you aware of how little of modern mathematics has been formalised? As in, properly formalised on a computer. Not just written up into a paper that other mathematics can read and nod along to.
Mathematics might seem very formal and serialised (and it is, compared to most other human endeavours) but that’s actually quite far from the truth. Really, it all exists in the mind of the mathematician and a lot of it is hard, if not currently impossible, to pin down precisely enough to enter into a formal system.
I think you probably do understand some things ‘transcendently’! Almost by definition they’re the things you’re least aware of understanding.
Experience is what's hard to square with computability. David Chalmer's calls this the hard problem. As long as you're taking about producing speech or other behaviors, it's easy to see how that might be a computation (and nothing more).
It's harder (for me) to see how it's possible to say that pain is just a way of describing things, i.e. that there's in principle no difference between feeling pain and computing a certain function.
Remember - there is no such thing as an objective consciousness meter.
Emulating the behaviours we associate with consciousness - something that still hasn't been achieved - solves the problem of emulation, not the problem of identity.
The idea that an emulation is literally identical to the thing it emulates in this instance only is a very strange belief.
Nowhere else in science is a mathematical model of something considered physically identical and interchangeable with the entity being modelled.
> Perhaps he and other true geniuses can understand things transcendently. Not so for me. My thoughts are serialized and obviously countable.
You needn't be a genius. Go on a few vipassana meditation retreats and your perception of all this may shift a bit.
> any kind of theorem or idea communicated to another mathematician needs to be serialized into language which would make it computable
Hence the suggestion by all mystical traditions that truth can only be experienced, not explained.
It may be possible for an AI to have access to the same experiences of consciousness that humans have (around thought, that make human expressions of thought what they are) - but we will first need to understand the parts of the mind / body that facilitate this and replicate them (or a sufficient subset of them) such that AI can use them as part of its computational substrate.
What is "understanding transcendently"? Just because Penrose is an authority on some subjects in theoretical physics doesn't mean he is a universal genius and that his ideas on consciousness or AI hold any value.
We gotta stop making infaillible super heroes/geniuses of people.
In this particular case, Penrose is a convinced dualist and his theories are unscientific. There are very good reasons to not be a dualist, a minority view in philosophy, which I would encourage anyone to seek if they want to better understand Penrose's position and where it came from.
He’s written and cowritten 5 books on the subject, going back nearly 40 years. I think he can as much as anyone can be considered an “authority” on something as inherently hard to observe or develop falsifiable theories as subjective conscious experience.
This isn’t an example of physicist stumbling into a new field for the first time and saying “oh that’s an easy problem. you just need to…”
The ideas of a very smart person who has spent decades thinking about a problem tend to be very valuable even if you don’t agree with them.
I watched half of the video. He keeps appealing to the idea that Goedel applies to AI because AI doesn't understand what it's doing. But I seriously doubt that we humans really know what we're doing, either.
IIRC, his Goedel argument against AI is that someone could construct a Goedel proposition for an intelligent machine which that machine could reason its way through to hit a contradiction. But, at least by default, humans don't base their epistemology on such reasoning, and I don't see why a conscious machine would either. It's not ideal, but frankly, when most humans hit a contradiction, they usually just ignore whichever side of the contradiction is most inconvenient for them.
The argument does not need to involve typical human behaviour with its faults. Because if a computer can simulate humans, it can also simulate humans with error correction and verification mechanisms. So, the computer should also be able to simulate a process where a group of humans write down initial deductions and then verify it extensively for logical errors using both computers and other humans.
Most of the objections have been covered in his book "Shadows of the Mind".
Also, the fact that most human behaviour is not about deducing theorems isn't relevant as that is used as a counterexample which attacks the 'computers can simulate humans' hypothesis. This particular behaviour is chosen, as it is easy to make reflective arguments precise.
This is not a presupposition for Penrose, but a conclusion. The argument for the conclusion is the subject of several of his books.
Secondly, the issue is not being a genius, but an ability to reflect. What can be shown, uncontroversially, is that a formal computer system which is knowably correct, a human (or indeed a machine apart which is not the original system) can know something(like a mathematical theorem) which is not accesible to the system. This is due to a standard diagonalization argument used in logic and computability.
The important qualifier is 'knowably correct' which doesn't apply to LLMs which are famous for their hallucinations. But, this is not a solid argument for LLMs being able to do everything that humans can do. Because correctness need not refer to immediate outputs, but outputs which are processed through several verification systems.
would you mind summarizing what the main argument is? I've watched several of his interviews (but not read the books), and I don't really understand why he concludes that consciousness is not computable.
Penrose does not require transcendent insights, but merely the ability to examine a finite knowably correct system to arrive at a correct statement which is not provable by the system. In fact, the construction of a Godel statement is mechanical, but this does not mean that the original system can see it. It is a bit like given a supposed finite list of all primes, we can construct a new prime by multiplying them and adding 1. This construction is a simple computation.
He sets up a definition where "real intelligence" requires consciousness, then argues AI lacks consciousness, therefore AI lacks real intelligence. This is somewhat circular.
The argument that consciousness can't be computable seems like a stretch as well.
Consciousness is not a result, it cannot be computed. It is a process, and we don't know how it interacts with computation. There are only two things I can really say about consciousness, and both are speculation: I think it isn't observable, and I think it is not a computation. For the first point, I can see no mechanism by which consciousness could affect the world so there is no way to observe it. For the second, imagine a man in a vast desert filled only with a grid of rocks that have two sides, a dark and light side and he has a small book which gives him instructions on how to flip these rocks. It seems unlikely that the rocks are sentient, yet certain configurations of rocks and books could produce the thought computation of the human mind. When does the sentience happen? If the man flips only a single rock according to those rules, would the computer be conscious? I doubt it. Does the consciousness exist between the flips of rock when he walks to the next stone? The idea that computation creates consciousness seems plainly untenable to me.
Indeed, I also think consciousness cannot be reduced to computation.
Here is one more thing to consider. All consciousness we can currently observe is embodied; all humans have a body and identity. We can interact with separate people corresponding to separate consciousnesses.
But if computation is producing consciousness, how is its identity determined? Is the identity of the consciousness based on the set of chips doing the computation? It is based on the algorithms used (i.e., running the same algorithm anywhere animates the same consciousness)?
In your example, if we say that consciousness somehow arises from the computation the man performs itself, then a question arises: what exactly is conscious in this situation? And what are the boundaries of that consciousness? Is the set of rocks as a whole? Is it the computation they are performing itself? Does the consciousness has a demarcation in space and time?
There are no satisfying answers to these questions if we assume mere computation can produce consciousness.
I think to argue usefully about consciousness you've got to be able to define what you mean by it. If you use in the sense of a boxer is knocked unconscious as he's not aware of anything much versus conscious where he knows what's going on and can react and punch back, then AI systems can also be aware or not and react or not.
If you say it's all about the feelings and machines can't feel that way then it gets rather vague and hard to reason about. I mean they don't have much in the way of feelings now but I don't see why they shouldn't in the future.
I personally feel both those aspects of consciousness are not woo but the results mechanisms built by evolution for functional purposes. I'm not sure how they could have got their otherwise unless you are going to reject evolution and go for divine intervention or some such.
Penrose believes that consciousness originates from quantum mechanics and the collapse of the wavefunction. Obviously you couldn't (effectively) simulate that with a classical computer. It's a very unconventional position, but it's not circular.
This is just another form of "god of the gaps" - Penrose desperately wants an interpretation that would allow for freedom of will, and so he constructs a theory around a physical process that allows for randomness, despite there being no evidence that this process is actually relevant to consciousness.
The fundamental result of Gödel's theorem is that logical completeness and logical consistency are complimentary; if a logical system has consistent rules then it will contain statements that are unprovable by the rules but true nonetheless, so it is incomplete. Alternately, if there is a proof available for all true statements via the rules then the rules used are inconsistent.
I think this means that "AGI" is limited as we are. If we build a machine that proves all true statements then it must use inconsistent rules, implying it is not a machine we can understand in the usual sense. OTOH, if it is using consistent rules (that do not contain contradiction) then it cannot prove all true statements so it ia not generally intelligent, but we can understand how it works.
I agree with Dr. Penrose about the misnomer of "artificial Intelligence". We ought to be calling the current batch of intelligence technologies "algabreic intelligence" and admiting that we seek "geometric intelligence" and have no idea how to get there.
The issue isn't the mere existence of two thinking modes (algebraic vs. geometric), but that we’ve culturally prioritized and trained mostly algebraic modes (linear language, math, symbolic logic). This has obscured our natural geometric capacity, especially the neurological pathways specialized in visually processing and intuitively understanding phenomena, particularly light itself (photons, vision, direct visual intuition of physics). Historically, algebraic thinking was elevated culturally around the Gnostic period (200 BCE forward), pushing aside the brain's default "geometric mode". Heck, during that period of history, people actively and forcefully campaigned against overly developing the analytical use of the mind. We should be actively mapping neurological pathways specialized for direct intuitive visual-physical cognition (understanding light intuitively at a neurological level, not symbolically or algebraically) for that to happen. Also: Understanding or explainability is not directly linked to consistency in the logical sense. A system can be consistent yet difficult to fully understand, or even inconsistent yet still partially understandable. We are talking right now here because we were put here through a series of historical events. Go back to 200 BCE and play out the Gnostic or Valentinus path to 2025.
When I think about understanding, in principle I require consistency not completeness. In fact, understandability is predicated on consistency in my view.
If I liken the quest for AGI to the quest for human flight, wherein we learned that the shape of the wing provides nearly effortless lift, while wing flapping only provides a small portion of the lift for comparatively massive energy input, then I suspect we are only doing the AGI equivalent of wing flapping at this point.
Good question, perhaps its best to start with what I mean by algabreic intelligence, then the contrast will be more clear. Algabreic intelligence uses the simple idea of equality to produce numerical unknowns from the known via standard mechanistic operations. So algabreic intelligence is mechanistic, operational, deductive, and quntitative. In contrast, geometric intelligence is concerned with the higher level abstract concepts of congruity, scale.
To return to my previous analogy, algabreic intelligence is wing flapping while geometric intelligence is the shape of the wing. The former is arduous time consuming and energy inefficient while the latter is effortless, and unreasonably effective.
I complement Penrose for his indifference to haters and harsh skeptics.
Our minds and consciousness do not fundamentally use linear logic to arrive at their conclusions, they use constructive and destructive interference. Linear logic is simulated upon this more primitive (and arguably superior) cognition.
It is true that any outcome of any process may be modeled in serialized terms or computational postulations, this is different than the interference feedback loop used by intelligent human consciousness.
Constructive and destructive interference is different and ultimately superior to linear logic on many levels. Despite this, the scalability of artificial systems may very well easily surpass human capabilities on any given task. There may be an arguable energy efficiency angle.
Constructive/destructive interference builds holographic renderings which work sufficiently when lacking information. A linear logic system would simulate the missing detail from learned patterns.
Constructive/destructive interference does not require intensive computation
An additive / reduction strategy may change the terms of a dilemma to support a compromised (or alternatively superior) “human” outcome which a logic system simply could not “get” until after training.
There is more, though these are a worthy start.
And consciousness is the inflection (feedback reverberation if you like) upon the potential of existential being (some animate matter in one’s brain). The existential Universe (some part of matter bound in the neuron, those micro-tubes perhaps) is perturbed by your neural firings. The quantum domain is an echo chamber. Your perspectives are not arranged states, they are potentials interfering.
Also, “you all” get intelligence and “will” wrong. I’ll pick that fight on another day.
I swear this was on the front page 2 minutes ago and now it’s halfway down page 2.
Anyway, I’m not really sure where Penrose is going with this. As a summary, incompleteness theorem is basically a mathematical reformulation of the paradox of the liar - let’s state this here for simplicity as “This statement is a lie” which is a bit easier than talking about “ All Cretans are liars”, which is the way I first heard it.
So what’s the truth value of “This statement is a lie”? It doesn’t have one. If it’s false, then it’s true. But if it’s true, then it must be false. The reason for this paradox is that it’s a self-referential statement: it refers to its own truth value in the construction of its own truth value, so it never actually gets constructed in the first place.
You can formulate the same sort of idea mathematically using sets, which is what Gödel did.
Now, the thing about this is that as far as I am aware (and I’m open to be corrected on this) this never actually happens in reality in any physical system. It seems to be an artefact of symbolic representation. We can construct a series of symbols that reference themselves in this way, but not an actual system. This is much the same way as I can write “5 + 5 = 11” but it doesn’t actually mean anything physically.
The closest thing we might get to would be something that oscillates between two states.
We also ourselves, don’t have a good answer to this problem as phrased. What is the truth value of “This statement is a lie”? I have to say “I don’t know” or “there isn’t one” which is a bit like cheating. Am I incapable of consciousness as a result? And if I am indeed conscious instead because I can make such a statement instead of simply ”True” or “False”, well I’m sure that an AI can be made to do likewise.
So I really don’t think this has anything to do with intelligence, or consciousness, or any limits on AI.
(for the record, I think the Penrose take on Gödel and consciousness is mostly silly and or confused)
I think your understanding of the incompleteness theorem is a little, well, incomplete. The proof of the theorem does involve, essentially, figuring out how to write down "this statement is not provable" and using liar-paradox-type-reasoning to show that it is neither provable nor disprovable.
But the incompleteness theorem itself is not the liar paradox. Rather, it shows that any (consistent) system rich enough to express arithmetic cannot prove or disprove all statements. There are things in the gaps. Gödel's proof gives one example ("this statement is not provable") but there are others of very different flavors. The standard one is consistency (e.g. Peano arithemtic alone cannot prove the consistency of Peano arithmetic, you need more, like much stronger induction; ZFC cannot prove the consistency of ZFC, you need more, like a large cardinal).
And this very much does come up for real systems, in the following way. If we could prove or disprove each statement in PA, then we could also solve the halting problem! For the same reason there's no general way to tell whether each statement of PA has a proof, there's no general way to tell whether each program will halt on a given input.
Nice reply. I don’t know anything about Peano arithmetic, or how it applies to the halting problem, so I can’t really evaluate this. All I know is the description of the proof that I read some time ago. Maybe there’s more to dig into on it, but as you say at the start of your post, likely none of it has anything to do with what Penrose is arguing for.
Penrose is saying a computer cannot understand / has no intelligence, and the incompleteness theorem proves this. Think in terms of the semantics (the meaning) of what the AI is outputting. The incompleteness theorem proves it does not understand and can never understand the semantics. It operates at the syntax level. How can something operating at the syntax level be intelligent? All it can do is look at symbols and rearrange them. There are algorithms to categorize and give these symbols (tokens/words) weights etc but all it can do is rearrange symbols. Do humans re-arrange symbols, no, they understand. They experience the color red, but to a computer the color red is a number. What is the connection between the number for red and the color red? AI pretty much collects every instance of red and in some sense approximates what red actually is.
Goedel's theorem is only a problem if you assume that intelligence is complete. (where complete means: able to determine whether any formal statement is true or false). We know that anything running on a computer is incomplete (e.g. Turing halting problems). For any of this to be interesting, Penrose would have to demonstrate that human intelligence is complete in some sense of the word. This seems highly unlikely. Superficially, human intelligence is not remotely complete since it is frequently unable to answer questions that have yes or no answers, and even worse, is frequently wrong. So not complete, either.
Readit News
We think, that is a fact.
Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.
Now, the question is: can we create a smaller function capable of performing the same feat?
If we assume that that function is computable in the Turing sense then, kinda yes, there are an infinite number of turing machines that given enough time will be able to produce the expected results. Basically we need to find something between our own brain and the Kolmogorov complexity limit. That lower bound is not computable, but given that my cats understands when we are discussing to take them to the vet then... maybe we don't really need a full sized human brain for language understanding.
We can run Turing machines ourselves, so we are at least Turing equivalent machines.
Now, the question is: are we at most just Turing machines or something else? If we are something else, then our own CoT won't be computable, no matter how much scale we throw at it. But if we are then it is just matter of time until we can replicate ourselves.
When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs. So human thinking is probably computable, and I think that LLMs can be said to be ”think” in ways that are analogous to what we do.
But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way.
I don’t necessarily think that you need to subscribe to dualism or religious beliefs to explain consciousness - it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”.
But I do think it’s still important to maintain a distinction between “thinking” (computable, we do it, AIs do it as well) and “consciousness” (we experience it, probably many animals experience it also, but it’s orthogonal to the linguistic or logical reasoning processes that AIs are currently capable of).
At some point this vague experience of awareness may be all that differentiates us from the machines, so we shouldn’t dismiss it.
> You've got to be careful when you say what the human does, if you add to the actual result of his effort some other things that you like, the appreciation of the aesthetic... then it gets harder and harder for the computer to do it because the human beings have a tendency to try to make sure that they can do something that no machine can do. Somehow it doesn't bother them anymore, it must have bothered them in earlier times, that machines are stronger physically than they are...
- Feynman
https://www.youtube.com/watch?v=ipRvjS7q1DI
Function can mean inputs-outputs. But it can also mean system behaviors.
For instance, recurrence is a functional behavior, not a functional mapping.
Similarly, self-awareness is some kind of internal loop of information, not an input-output mapping. Specifically, an information loop regarding our own internal state.
Today's LLMs are mostly not very recurrent. So might be said to be becoming more intelligent (better responses to complex demands), but not necessarily more conscious. An input-output process has no ability to monitor itself, no matter how capable of generating outputs. Not even when its outputs involve symbols and reasoning about concepts like consciousness.
So I think it is fair to say intelligence and consciousness are different things. But I expect that both can enhance the other.
Meditation reveals a lot about consciousness. We choose to eliminate most thought, focusing instead on some simple experience like breathing, or a concept of "nothing".
Yet even with this radical reduction in general awareness, and our higher level thinking, we remain aware of our awareness of experience. We are not unconscious.
To me that basic self-awareness is what consciousness is. We have it, even when we are not being analytical about it. In meditation our mind is still looping information about its current state, from the state to our sensory experience of our state, even when the state has been reduced so much.
There is not nothing. We are not actually doing nothing. Our mental resting state is still a dynamic state we continue to actively process, that our neurons continue to give us feedback on, even when that processing has been simplified to simply letting that feedback of our state go by with no need to act on it in any way.
So consciousness is inherently at least self-awareness in terms of internal access to our own internal activity. And that we retain a memory of doing this minimal active or passive self-monitoring, even after we resume more complex activity.
My own view is that is all it is, with the addition of enough memory of the minimal loop, and a rich enough model of ourselves, to be able to consider that strange self-awareness looping state afterwards. Ask questions about its nature, etc.
I imagined the Catholic Church, for example, would be publishing missives reminding everyone that only humans can have souls, and biologists would be fighting an quixotic battle to claim that consciousness can arise from physical structures and forces.
I'm still surprised at how credulous and accepting societies have been of AI developments over the last few years.
I've heard this idea before but I have never been able to make head or tail of it. Consciousness can't be an illusion, because to have an illusion you must already be conscious. Can a rock have illusions?
"We've all been dancing around the basic issue: does Data have a soul?" -- Captain Louvois. https://memory-alpha.fandom.com/wiki/The_Measure_Of_A_Man_(e...
It likely is a fact, but we don't really know what we mean by "think".
LLMs have illuminated this point from a relatively new direction: we do not know if their mechanism(s) for language generation are similar to our own, or not.
We don't really understand the relationship between "reasoning" and "thinking". We don't really understand the difference between Kahneman's "fast" and "slow" thinking.
Something happens, probably in our brains, that we experience and that seems causally prior to some of our behavior. We call it thinking, but we don't know much about what it actually is.
AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain.
The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them.
When will it be able to set its own goals, and know when it has achieved them?
When will it be able to recognize that it doesn't know something and do the work to fill in the blanks.
I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.
I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.
We mistakenly assume, they are true because perhaps we want them to be true. But we have no proof that either of these are true.
As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Is there a Turing machine that can output all the free parameters of the standard model?
Sure, just hard code them.
> As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values.
Things seem to be quantised at a low enough level.
Also: interestingly enough quantum mechanics is both completely deterministic and linear. That means even if it was continuous, you could simulate it to an arbitrary precision without errors building up chaotically.
(Figuring out how chaos, as famously observed in the weather, arises in the real world is left as an exercise to the reader. Also a note: the Copenhagen interpretation introduces non-determinism to _interpret_ quantum mechanics but that's not part of the underlying theory, and there are interpretations that have no need for this crutch.)
And don't forget Church-Turing, Gödel Numbers, and all the other stuff. Programming is math and Gödel did essential work on the theory of computation. It would be weird NOT to include his work in this conversation.
But this is a great question. Many believe no. Personally I'm unsure, but lean no. Penrose is a clear no but he has some wacky ideas. Problem is, it's hard to tell a bad wacky idea from a good wacky idea. Rephrasing Clarke's Second Law: Genius is nearly indistinguishable from insanity. The only way to tell is with time.But look into things like NARS and Super Turing machines (Hypercomputation). There's a whole world of important things that are not often discussed when it comes to the discussion of AGI. But for those that don't want to dig deep into the math, pick up some Sci-Fi and suspend your disbelief. Star Trek, The Orville and the like have holographic simulations and I doubt anyone would think they're conscious, despite being very realistic. But The Doctor in Voyager or Isaac in The Orville are good examples of the contrary. The Doctor is an entity you see become conscious. It's fiction, but that doesn't mean there aren't deep philosophical questions. Even if they're marked by easy to digest entertainment. Good stories are like good horror, they get under your skin, infect you, and creep in
Edit:
I'll leave you with another question. Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
A Turing machine can in principle beat the Turing test. But so can a giant lookup table, if there's any finite time limit (however generous) placed on the test.
The 'magic' would in the implementation of the table (or the Turing machine) into something that can answer in a reasonable amount of time and be physically realised in a reasonable amount of space.
Btw, that's an argument from Scott Aaronson's https://www.scottaaronson.com/papers/philos.pdf
In addition to the detectability problem, I wrote in the adjacent comment, this question can be further refined.
A Turing machine is an abstract concept. Do we need to take into account material/organizational properties of its physical realization? Do we need to take into account computational complexity properties of its physical realization?
Quantum mechanics without Penrose's Orch OR is Turing computable, but its runtime on classical hardware is exponential in, roughly, the number of interacting particles. So, theoretically, we can simulate all there is to simulate about a given person.
But to get the initial state of the simulation we need to either measure the person's quantum state (thus losing some information) or teleport his/her quantum state into a quantum computer (the no-cloning theorem doesn't allow to copy it). The quantum computer in this case is a physical realization of an abstract Turing machine, but we can't know its initial state.
The quantum computer will simulate everything there are to simulate, but the interaction of a physical human with the initial state of the Universe via photons of the cosmic microwave background. Which may deprive the simulated one of "free will" (see "The Ghost in the Quantum Turing Machine" by Scott Aaronson). Or maybe we can simulate those photons too, I'm not sure about it.
Does all of it have anything to do with consciousness? Yeah, those are interesting questions.
Another question. How do you go about detecting whether consciousness has arisen?
Simultaneously, actual dualists flock to his theories because they think association with Penrose lends credibility to their religious stuff.
To pull from the relevant part of Hofstadter’s incredible I am a Strange Loop (a book also happens to more rigorously invoke Gödel for cognitive science):
Highly recommend it for anyone who liked Gödel, Escher, Bach, but wants more explicit scientific theses! He basically wrote it to clarify the more artsy/rhetorical points made in the former book.It’s possible that we haven’t found a way to express your thinking function digitally, which I think is true, but I have a feeling that the complexity of thought requires the analog-ness of our brains.
Other aspects of ANN that show that Gödel doesn’t apply is that they are not formal systems. Formal system is a collection of defined operations. The building blocks of ANN could perhaps be built into a formal system. Petri nets have been demonstrated to be computationally equivalent to Turing machines. But this is really an indictment on the implementation. It’s the same as using your PC, implementing a formal system like its instruction set to run a heuristic computation. Formal system can implement informal systems.
I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.
As for humans, there is no way you can look at the behavior of a human and know for certain it is not a Turing machine. With a large enough machine, you could simulate any behavior you want, even behavior that would look, on first observation, to not be coming from a Turing machine; this is a form of the halting problem. Any observation you make that makes you believe it is NOT coming from a Turing machine could be programmed to be the output of the Turing machine.
Incorrect.
The comment above confuses some concepts.
Perhaps this will help: consider a PRNG implemented in software. It is an algorithm. The question of the utility of a PRNG (or any algorithm) is a separate thing.
It is a self-delimiting program. It is an algorithm in the most basic sense of the definition of “partial recursive function” (total in this case) and thus all known results of computability theory and algorithmic information theory apply.
> Formal system is a collection of defined operations
Not at all.
> I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.
We have zero evidence of this one way or another.
—
I’m looking for loopholes around Gödel’s theorems just as much as everyone else is, but this isn’t it.
Physicists like to use mathematics for modeling the reality. If our current understanding of physics is fundamentally correct, everything that can possibly exist is functionally equivalent to a formal system. To escape that, you would need some really weird new physics. Which would also have to be really inconvenient new physics, because it could not be modeled with our current mathematics or simulated with our current computers.
So if Gödel tells us that either formal systems will be consistent and make statements they cannot prove XOR be inconsistent and therefore unreliable, at least to some degree, then surely informal systems will, at best, be the same, and, at worst, be much worse?
"Thinked information" is a colour not an inherent property of information. The fact that information has been thought is like the fact it is copyrighted. It is not something inherent to the information, but a property of its history.
https://ansuz.sooke.bc.ca/entry/23
This is a big assumption. I'm not saying it's wrong, but I am saying it's not reasonable to just handwave and assume that it's right.
Where does this question come from? Especially where does the 'smaller' requirement come from?
You’re arguing that we know artificial reasoning exists because we are capable of reasoning. This presupposes that reasoning is computable and that we ourselves reason by computation. But that’s exactly what Penrose is saying isn’t the case - you’re saying we’re walking Turing machines, we’re intelligent, so we must be able to effectively create copies of that intelligence. Penrose is saying that intelligence is poorly defined, that it requires consciousness which is poorly understood, and that we are not meat-based computers.
Your last question misses the point completely. “If we are something else, then out CoT won’t be computable…” it’s like you’re almost there but you can’t let go of “we are meat-machines, everything boils down to computation, we can cook up clones”. Except, “basic computability theory” says that’s not even wrong.
What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false
No this isn't the checkmate you think it is. It could still be a mathematical function. But every person transforming information into "thinked information" could have a different implementation of this function. Which would be expected as no person is made of the same code (DNA).
FWIW human brain does indeed consume a lot of energy, accounting for over 20% of our body metabolism. We don't know how to attribute specific watts consumed to specific thoughts because we don't understand the functioning of the brain enough, but there's no obvious reason why it shouldn't be possible.
Dead Comment
1. I don't think human reasoning is consistent in the technical sense, which makes the incompleteness theorem inapplicable regardless of what you think about us and Turing machines.
2. The human brain is full of causal cycles at all scales. Even if you think human reasoning is axiomatisable, it's not at all obvious to me that the set of axioms would be finite or even computable. Again this rules out any application of Gödel's theorem.
3. Penrose's argument revolves around the fact that the sentence encoding "true but not provable" in Gödel's argument is actually provably true in the outer logical system being used to prove Gödel's theorem, just not the inner logical system being studied. But as all logicians know, truth is a slippery concept and is itself internally indefinable (Tarski's theorem), so there's no guarantee that this notion of "truth" used in the outer system is the same as the "real" truth predicate of the inner system (at best it's something like an arbitrary choice, dependent on your encoding). Penrose is referring to "truth" at multiple logical levels and conflating them.
In other words: you can't selectively chose to apply Gödel's theorem to the situation but not any of the other results of mathematical logic.
I don't understand how you mean (3) to apply as a criticism at all. He is not making a claim about truth at some level, he's just reminding us what computation is and what its limits are.
afaiu, there are two ways to counter Penrose's claim:
a. Prove that consciousness is actually just computational.
b. Prove that merely stacking computations can somehow produce a super-computational system.
1. Assume for contradiction that human reasoning is formally describable, as an algorithm or as a logical structure.
2. As a result, the incompleteness theorem produces a sentence that is true in that formal system but which human reasoning cannot detect (by virtue of being described by this formal system).
3. But the proof of the incompleteness theorem shows that this sentence is true, and it was produced by a human, a contradiction.
I don't necessarily disagree with the conclusion (I'm kinda agnostic at this point), I just think that this particular argument doesn't hold water.
Deleted Comment
Everything that makes "truth" slippery makes "intelligence" and "consciousness" even more slippery and subjective. This is why AGI has such negative impact on AI discourse -- the cause only advances when we focus on improving at measurable tasks.
I like to think people can still make progress on questions of intelligence and consciousness though. Michael Levin's work comes to mind, for instance. Science is just at very early stages of understanding =)
The reasoning is representable with and by a finite number of elementary physical particles and so must itself be finite. Because it is finite it is computable.
Said another way, you would need an infinitely large brain (or an infinitely deep one) to create infinite reasoning.
Busy Beaver numbers are finite, but not computable.
Do we even know this much?
Human reasoning is certainly limited. I mean, imagine the kinds of formulas one would get applying GT to the brain. They would be so enormous they'd be entirely impenetrable to human reasoning. We couldn't even read them in a single lifetime.
As for proving GT itself, this proof has been formalized. There's no reason a computer couldn't prove GT by itself.
For instance the set of even numbers is infinite but computable, just check whether the number is divisible by 2.
The algorithm itself is finite, even if the set it determines is not.
I think the liar's paradox is of a different kind. It's a sentence that looks well-formed but arguably has no truth value. If you were to formalise human reasoning as a logical system, such sentences would not be definable in it.
Either way, for Penrose's argument to work you actually need the proof of Gödel's theorem to hold, not just the result.
Perhaps he and other true geniuses can understand things transcendently. Not so for me. My thoughts are serialized and obviously countable.
And in any case: any kind of theorem or idea communicated to another mathematician needs to be serialized into language which would make it computable. So I’m not convinced I could be convinced without a computable proof.
And finally just like computable numbers are dense in the reals, maybe computable thoughts are dense in transcendence.
His intent at the time was to open a physical explanation for free will by taking the recourse to quantum nano-tubules magnifying true randomness to the level of human cognition. As much as I'm also skeptical that this actually moves the needle on whether or not we have free will (...vs occasionally having access to statistically-certain nondeterminism? Ok...) the computable stuff was just in service of this end.
I strongly suspect he just hasn't grasped how powerful heuristics are at overcoming general restrictions on computation. Either that or this is an ideological commitment.
Kind of sad—penrose tilings hold a special place in my heart.
Free will is a useful abstraction. Just like life and continuity of self are.
> I strongly suspect he just hasn't grasped how powerful heuristics are at overcoming general restrictions on computation.
Allowing approximations or "I don't know" is what's helpful. The bpf verifier can work despite the halting problem being unsolvable, not because it makes guesses (uses heuristics) but because it's allowed to lump in "I don't know" with "no".
That's Penrose's old criticism. We're past that. It's the wrong point now.
Generative AI systems are quite creative. Better than the average human at art. LLMs don't have trouble blithering about advanced abstract concepts. It's concrete areas where these systems have trouble, such as arithmetic. Common sense is still tough. Hallucinations are a problem. Lying is a problem. None of those areas are limited by computability. It's grounding in the real world that's not working well.
(A legit question to ask today is this: We now know how much compute it takes to get to the Turing test level of faking intelligence. How do biological brains, with such a slow clock rate, do it? That was part of the concept behind "nanotubules". Something in there must be running fast, right?)
If so then it really comes down to believing something not because you can prove it but because it is true.
I’m just a mediocre mathematician with rigor mortis. So I won’t be too hard on Penrose.
This is a fallacy. Just because you need to serialize a concept to communicate it doesnt mean the concept itself is computable. This is established and well proven:
https://en.wikipedia.org/wiki/List_of_undecidable_problems
The fact that we can come up with this kind of uncumputable problems is a big plus in supprt of Penrose's Idea that consciousnes is not computable and goes way beyond compatability.
You might want to consider doing a bit of meditation...anyone who describes their thoughts as 'serialized' and 'obviously countable' has not much time actually looking at their thoughts.
Are you aware of how little of modern mathematics has been formalised? As in, properly formalised on a computer. Not just written up into a paper that other mathematics can read and nod along to.
Mathematics might seem very formal and serialised (and it is, compared to most other human endeavours) but that’s actually quite far from the truth. Really, it all exists in the mind of the mathematician and a lot of it is hard, if not currently impossible, to pin down precisely enough to enter into a formal system.
I think you probably do understand some things ‘transcendently’! Almost by definition they’re the things you’re least aware of understanding.
It's harder (for me) to see how it's possible to say that pain is just a way of describing things, i.e. that there's in principle no difference between feeling pain and computing a certain function.
i would be a bit more aggressive: Penrose asserts without evidence
Remember - there is no such thing as an objective consciousness meter.
Emulating the behaviours we associate with consciousness - something that still hasn't been achieved - solves the problem of emulation, not the problem of identity.
The idea that an emulation is literally identical to the thing it emulates in this instance only is a very strange belief.
Nowhere else in science is a mathematical model of something considered physically identical and interchangeable with the entity being modelled.
You needn't be a genius. Go on a few vipassana meditation retreats and your perception of all this may shift a bit.
> any kind of theorem or idea communicated to another mathematician needs to be serialized into language which would make it computable
Hence the suggestion by all mystical traditions that truth can only be experienced, not explained.
It may be possible for an AI to have access to the same experiences of consciousness that humans have (around thought, that make human expressions of thought what they are) - but we will first need to understand the parts of the mind / body that facilitate this and replicate them (or a sufficient subset of them) such that AI can use them as part of its computational substrate.
We gotta stop making infaillible super heroes/geniuses of people.
In this particular case, Penrose is a convinced dualist and his theories are unscientific. There are very good reasons to not be a dualist, a minority view in philosophy, which I would encourage anyone to seek if they want to better understand Penrose's position and where it came from.
This isn’t an example of physicist stumbling into a new field for the first time and saying “oh that’s an easy problem. you just need to…”
The ideas of a very smart person who has spent decades thinking about a problem tend to be very valuable even if you don’t agree with them.
IIRC, his Goedel argument against AI is that someone could construct a Goedel proposition for an intelligent machine which that machine could reason its way through to hit a contradiction. But, at least by default, humans don't base their epistemology on such reasoning, and I don't see why a conscious machine would either. It's not ideal, but frankly, when most humans hit a contradiction, they usually just ignore whichever side of the contradiction is most inconvenient for them.
Most of the objections have been covered in his book "Shadows of the Mind".
Also, the fact that most human behaviour is not about deducing theorems isn't relevant as that is used as a counterexample which attacks the 'computers can simulate humans' hypothesis. This particular behaviour is chosen, as it is easy to make reflective arguments precise.
Secondly, the issue is not being a genius, but an ability to reflect. What can be shown, uncontroversially, is that a formal computer system which is knowably correct, a human (or indeed a machine apart which is not the original system) can know something(like a mathematical theorem) which is not accesible to the system. This is due to a standard diagonalization argument used in logic and computability.
The important qualifier is 'knowably correct' which doesn't apply to LLMs which are famous for their hallucinations. But, this is not a solid argument for LLMs being able to do everything that humans can do. Because correctness need not refer to immediate outputs, but outputs which are processed through several verification systems.
If they prove it then they have either shown that the idea is not transcendent or that Gödel's theorum is false.
That's the same as saying "I know the answer, when you are speculating"
Yes. He has also written books about it.
https://en.wikipedia.org/wiki/Roger_Penrose#Consciousness
He has been very clear in that he claims exactly that and that there is Quantum Mechanics (wave function collapse in particular) involved.
I personally think he's probably wrong, but who really knows?
The argument that consciousness can't be computable seems like a stretch as well.
Here is one more thing to consider. All consciousness we can currently observe is embodied; all humans have a body and identity. We can interact with separate people corresponding to separate consciousnesses.
But if computation is producing consciousness, how is its identity determined? Is the identity of the consciousness based on the set of chips doing the computation? It is based on the algorithms used (i.e., running the same algorithm anywhere animates the same consciousness)?
In your example, if we say that consciousness somehow arises from the computation the man performs itself, then a question arises: what exactly is conscious in this situation? And what are the boundaries of that consciousness? Is the set of rocks as a whole? Is it the computation they are performing itself? Does the consciousness has a demarcation in space and time?
There are no satisfying answers to these questions if we assume mere computation can produce consciousness.
If you say it's all about the feelings and machines can't feel that way then it gets rather vague and hard to reason about. I mean they don't have much in the way of feelings now but I don't see why they shouldn't in the future.
I personally feel both those aspects of consciousness are not woo but the results mechanisms built by evolution for functional purposes. I'm not sure how they could have got their otherwise unless you are going to reject evolution and go for divine intervention or some such.
Where would the placebo effect fit in this thought experiment?
> a grid of rocks that have two sides, a dark and light side and he has a small book
Where did the book come from?
Dead Comment
https://en.wikipedia.org/wiki/The_Emperor%27s_New_Mind
https://en.wikipedia.org/wiki/Shadows_of_the_Mind
Deleted Comment
I think this means that "AGI" is limited as we are. If we build a machine that proves all true statements then it must use inconsistent rules, implying it is not a machine we can understand in the usual sense. OTOH, if it is using consistent rules (that do not contain contradiction) then it cannot prove all true statements so it ia not generally intelligent, but we can understand how it works.
I agree with Dr. Penrose about the misnomer of "artificial Intelligence". We ought to be calling the current batch of intelligence technologies "algabreic intelligence" and admiting that we seek "geometric intelligence" and have no idea how to get there.
When I think about understanding, in principle I require consistency not completeness. In fact, understandability is predicated on consistency in my view.
If I liken the quest for AGI to the quest for human flight, wherein we learned that the shape of the wing provides nearly effortless lift, while wing flapping only provides a small portion of the lift for comparatively massive energy input, then I suspect we are only doing the AGI equivalent of wing flapping at this point.
To return to my previous analogy, algabreic intelligence is wing flapping while geometric intelligence is the shape of the wing. The former is arduous time consuming and energy inefficient while the latter is effortless, and unreasonably effective.
Our minds and consciousness do not fundamentally use linear logic to arrive at their conclusions, they use constructive and destructive interference. Linear logic is simulated upon this more primitive (and arguably superior) cognition.
It is true that any outcome of any process may be modeled in serialized terms or computational postulations, this is different than the interference feedback loop used by intelligent human consciousness.
Constructive and destructive interference is different and ultimately superior to linear logic on many levels. Despite this, the scalability of artificial systems may very well easily surpass human capabilities on any given task. There may be an arguable energy efficiency angle.
Constructive/destructive interference builds holographic renderings which work sufficiently when lacking information. A linear logic system would simulate the missing detail from learned patterns.
Constructive/destructive interference does not require intensive computation
An additive / reduction strategy may change the terms of a dilemma to support a compromised (or alternatively superior) “human” outcome which a logic system simply could not “get” until after training.
There is more, though these are a worthy start.
And consciousness is the inflection (feedback reverberation if you like) upon the potential of existential being (some animate matter in one’s brain). The existential Universe (some part of matter bound in the neuron, those micro-tubes perhaps) is perturbed by your neural firings. The quantum domain is an echo chamber. Your perspectives are not arranged states, they are potentials interfering.
Also, “you all” get intelligence and “will” wrong. I’ll pick that fight on another day.
Anyway, I’m not really sure where Penrose is going with this. As a summary, incompleteness theorem is basically a mathematical reformulation of the paradox of the liar - let’s state this here for simplicity as “This statement is a lie” which is a bit easier than talking about “ All Cretans are liars”, which is the way I first heard it.
So what’s the truth value of “This statement is a lie”? It doesn’t have one. If it’s false, then it’s true. But if it’s true, then it must be false. The reason for this paradox is that it’s a self-referential statement: it refers to its own truth value in the construction of its own truth value, so it never actually gets constructed in the first place.
You can formulate the same sort of idea mathematically using sets, which is what Gödel did.
Now, the thing about this is that as far as I am aware (and I’m open to be corrected on this) this never actually happens in reality in any physical system. It seems to be an artefact of symbolic representation. We can construct a series of symbols that reference themselves in this way, but not an actual system. This is much the same way as I can write “5 + 5 = 11” but it doesn’t actually mean anything physically.
The closest thing we might get to would be something that oscillates between two states.
We also ourselves, don’t have a good answer to this problem as phrased. What is the truth value of “This statement is a lie”? I have to say “I don’t know” or “there isn’t one” which is a bit like cheating. Am I incapable of consciousness as a result? And if I am indeed conscious instead because I can make such a statement instead of simply ”True” or “False”, well I’m sure that an AI can be made to do likewise.
So I really don’t think this has anything to do with intelligence, or consciousness, or any limits on AI.
I think your understanding of the incompleteness theorem is a little, well, incomplete. The proof of the theorem does involve, essentially, figuring out how to write down "this statement is not provable" and using liar-paradox-type-reasoning to show that it is neither provable nor disprovable.
But the incompleteness theorem itself is not the liar paradox. Rather, it shows that any (consistent) system rich enough to express arithmetic cannot prove or disprove all statements. There are things in the gaps. Gödel's proof gives one example ("this statement is not provable") but there are others of very different flavors. The standard one is consistency (e.g. Peano arithemtic alone cannot prove the consistency of Peano arithmetic, you need more, like much stronger induction; ZFC cannot prove the consistency of ZFC, you need more, like a large cardinal).
And this very much does come up for real systems, in the following way. If we could prove or disprove each statement in PA, then we could also solve the halting problem! For the same reason there's no general way to tell whether each statement of PA has a proof, there's no general way to tell whether each program will halt on a given input.
It set off the flamewar detector. I've turned that off now.