I've always noticed that when I'm giving advice to someone or trying to help out, it always feels their problem is easier than whatever problem I have. As someone with some anxiety around things like calling some company to get something done or asking a random stranger for some help in a store, I would gladly do it if it was to help someone else (family member or friend). But when it's for me I find it harder.
I wonder how much psychologically we can be more confident and less anxious when we're doing something for others vs ourselves..
I imagine it has to do with vulnerability. When you are asking for something or sharing something, being turned down feels personal. When doing it for someone else, it's no big deal if they say no.
What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our ability to act intentionally.
So in my mind: even if you grant all the AI-naysayer's complaints about how LLMs aren't "actually" thinking, you can still believe that they will end up being a component in a system which actually "does" think.
Especially when modeling acting with intent. The ability to measure against past results and think of new innovative approaches seems like it may come from a system that may model first and then use LLM output. Basically something that has a foundation of tools rather than an LLM using MCP. Perhaps using LLMs to generate a response that humans like to read, but not in them coming up with the answer.
Either way, yes, its possible for a thinking system to use LLMs (and potentially humans piece together sentences in a similar way), but its also possible LLMs will be cast aside and a new approach will be used to create an AGI.
So for me: even if you are an AI-yeasayer, you can still believe that they won't be a component in an AGI.