Not to mention that it seems immoral to mandate what every single human being does with their lives.
I keep seeing people say “AGI” but no one will define it for me. It’s always “you know it when you see it” or “when software can do anything”.
I’m alright with “things will continue to advance” but people assume it’s a single event that happens, not a goal post that represents “the dream” that moves 1 step further away every 1 step we advance.
Just shouting random platitudes and religious grandiosity at people is just annoying.
https://towardsdatascience.com/beyond-the-turing-test-two-sc...
Another comparison actually is the GameStop stocks phenomenon. At least some of them expected such high returns (in some cases nearly infinite), that it would be fiscally insane to invest in anything but GME.
The payoff doesn't even need to be theoretically infinite. A sufficiently large finite payoff is enough. The real problem, as you say, is that these theoretically large payoffs are just pulled out of thin air (or someplace less savory...).
In the case of AGI, there is also the well known viewpoint that AGI will result in a huge negative payoff, as soon as the AGI realizes that it can use the atoms you are made of for something that, in its view, is more productive. AGI evangelists like the GP fail to factor that in as well. When such considerations are taken into account, the expected payoff from AGI is basically zero, even without factoring in the issue I described above.
[1] I'm not saying it isn't possible, just that I see no solid reason to think it is.
The logic is that AGI will save countless lives and arguably make EVERY job easier and better. So not developing it is utterly immoral. Doing anything else you are literally (not literally) killing people.