Only if you're naive. I guess most here are.
Governments are paranoid, particularly about losing control and influence over its subjects. This is expected behaviour.
The question isn’t if some would attempt these behaviors, but rather if we and our democratic structures empower those people or fail to constrain them.
538 democrasized the numbers that were the domain of political whizzes. I don't know if that's a good thing.
https://cash.app/taxes is already free for both federal and state and handled all of my complicated tax needs the past two years.
"Blindly following instructions from an LLM would have killed me."
Not exactly shocking if you take into consideration that at the core, they're simply number predictors.
There are almost never company-killing fines. There are almost never consent decrees with stipulations that change the company's behavior. There are almost never instances of piercing the corporate veil and going after executives or shareholders. It's a total joke.
Whereas, settling meets the company's incentives (eliminating uncertainty), meets the regulator's incentives (bad behavior is stopped locally). The moral hazard created by making fraud seem less risky (because the punishments aren't that bad) is born by the public.
The solution here would be to limit the possible legal shenanigans that companies can use to increase the cost of taking a case to trial.
I thought this was still up in the air.