Even in the best of cases, our perceptions and interpretations of those perceptions are often dramatically flawed. Add on top of that neural atrophy, loss of oxygen, accumulation of senescent cells, poor clearance of waste products ... and what comes out is often going to be gibberish, as the remaining working parts of the mind attempts to construct a coherent narrative from the broken fragments of sensory input and failing memories.

I got interested in consciousness 35 years ago or so when I read Oliver Sacks' "The Man Who Mistook His Wife For A Hat". Although the various people depicted in the stories had some physical deficit due to a trauma of some kind, it vividly demonstrated how we rationalize our way through the world more than we reason our way through it. Our conscious mind is much more of a facade than we typically imagine.

When people say, "Well, LLMs are just generating token N+1 from the previous N tokens, they really aren't thinking", I counter with this: we have been having this discussion -- are you at all aware of the stream of words coming out of your mouth, or are you hearing them the same time that I am?

Yes, sometimes we have deliberate thought where we rehearse different lines of reasoning before uttering something, but 98% of the time we are spewing just like LLMs do. And when we do engage in deliberate thought, each of those trial sentences again just appears without consideration; we are simply post-hoc picking the one that feels best.