Because if you don't trust it, you're fucked:
- Your attempts to limit its influence on your life will be effective only if the ASI decides to willfully ignore or not notice you.
- If it lies or not, there's nothing you can really do. The outcome it wants is practically guaranteed regardless. It will be running circles around your mental capacity like a human drawing circles around an ant on a piece of paper.
So what option do you really have but to trust it? I mean, sure, you can not trust the all-powerful god, but your lack of trust will never really have any effect on the real world, or even your life, as the ASI will always get what it wants.
Really, your only option is to prevent it from happening to begin with.
All that said - I think ASI will be great and people's concerns are overblown.
That's a huge assumption.
AI in SF: all-powerful all-controlling abstract entity.
AI in reality: LLMs on thousands of servers with GPUs, operated and controlled by engineers, gated by a startup masquerading as a non-profit and operating for years at a loss, with a web interface and an API, fragile enough that it becomes unavailable at times.
I don't understand this sentence. I don't trust Generative AI because it often spits out false, inaccurate or made up answers, but I don't believe my "only option is to prevent it from existing".