On the other hand, not developing AGI puts you at the risk that your enemy will, so it's not really a choice, it must be done or else.
The real problem, as I see it, is that once AGI is achieved, and robotics is up to par, human work is not needed anymore, which puts most people in a strange position, something that never happened before, useless for people in power. We were never in this particularly strange position, historically speaking.
And I do not believe people who are looking to get AGI's power, and remove dependence on humans, are objective in their ideas about what must be done. Thus their thoughs should always be taken with a grain of salt.
The only out I see for most people to stay alive, post AGI powered robotics, is if AGI completely takes power and control from the hands of the people up top. Else the people in power will have a very dark incentive, which I believe will inevitably (sooner or later) result in a massive population loss across Earth.
I'd rather risk AGI's conclusions than psychos in power starting to see me as a "useless eater". The latter has a guaranteed outcome.
I personally feel so little connection to this ideology, especially in the post-COVID world. Wanting things my neighbor has occupies nearly zero space in my mind. I drive a 10 year old car that was not fancy when I bought it, and although I could afford to buy a new one, even a fancy one, I just. don’t. care.
Freedom is the only thing that motivates me, and Thiel’s worldview sounds like slavery. His totalitarianism is an inability to personally privatize more of the world than his neighbor. My totalitarianism is obligation to participate in that world.
What is the incentive of someone who thinks has a shot at having control over such powerful technology? They will do and say anything that is necessary to justify doing it.
My argument is simpler, if "we" don't do it "they" will do it. It's something we'll have to wing. It is surely happening. The real problem will come from the people having control over such tech, since the tech breaks an old game, as old as humans existed and started forming groups.
And I see no billionaire who wants to develop and take control over this tech, explain how they'll fix the problem it creates, at game level. I do not care about promises, they mean nothing. I only care about game strategy that guarantees most humans stay alive in such an environment, with age old game rules breaking down.