What's your suggested alternative? It's really easy to criticize, but a lot harder to do any better. If you aren't suggesting any other methods, you're implicitly saying we should just ignore the problem. I really don't think that will end well.

My suggested alternative is prioritizing problems appropriately. This "Open Philantropy Project" is a guy taking money that used to be for curing malaria (and I respect what he used to do on that front!) and giving it to his roommate.

You can do better solving a terrible problem that does exist than solving a sci-fi problem that doesn't exist.

I know it's usually uncouth to compare different charities like this, but that's exactly what Effective Altruism was supposed to be about, and this cause directly competes with curing malaria.

If more than zero AGI technology starts existing, we can re-prioritize. It makes no sense for a field to go from not existing to superhuman performance without anyone noticing.

Nobody thinks we won't notice when it's more imminent; the concern is that capability research has always been faster than safety research. If we wait until it's imminent, we risk having no chance of catching up in time. Why not devote a tiny fraction of our funding to starting the research now, so we can be ready on time? Worst case we're early... which in this context is a lot better than being late.

Worst case the safety research ends up causing the AGI to be actively antagonistic. Slightly less worst case it never leads anywhere and AGI ends up turning us into mouthless slugs anyway. Third-ish worst it makes AGI suborn-able and whoever gets to it first becomes our god-king.

many many worse situations later we get to:

It never leads anywhere but AGI never happens anyway and it just ends up being welfare-for-future-phobes instead of curing debilitating disease or buying every homeless person in a city a new pair of shoes or whatever.

"we're early" is in fact the best possible scenario, actually it's basically the only positive scenario.

You admit that you don't know what the actual probabilities are. Or you admit you have an extremely low confidence in your estimate of the probabilities.

Have you read any of the estimates? Basically every expert gives a huge confidence interval, as do most of the people actively working on AI risk.

Saying you are 95% confident AGI will come between 50-100 years (a "huge" confidence interval), is still far too confident.

I don't recall seeing 95% confidence intervals that narrow. Even the 90% confidence intervals I see are usually like 5-150 or something. (Also it would be super weird to me if the lower bound of someone's 95%CI was 50.)