You have to weigh the cost and the risk. Here the risk, how unlikely it might be, should warrant some extra preparation.
Let's instead look at the risks of boarding a plane. There's a very small chance that when you board a flight, instead of a plane that will fly you to your destination safe and sound, you're boarding a Flying Death Trap that will crash and burn, taking everybody onboard it to their deaths.
The chances of boarding an FDT is very small, infinitisemal. The cost however may as well be infinite- if you are killed, it's game over, no more rewards, no way to recoup the cost.
What is the rational behaviour then? To not board your flight, because if you do board an FDT you will certainly die and pay an infinite cost? Most people -if they consider the question at all- seem to think that if the chance of paying X cost is really small, it doesn't matter how large X is.
So people keep boarding their flights, not knowing until the last moment whether they're on a plane or an FDT. Some do indeed board FDTs and die in aviation accidents- rarely, but they do.
The article however says that they shouldn't. Since there is maximal uncertainty at the point where a flight is boarded (you can't know whether it's a plane or an FDT until the very last moment) you shouldn't be boarding. You shouldn't fly. At all. Because there's a tiny chance you might die.
Is that a better analogy?
And I'm pretty sure I do understand what he's talking about. He's saying that the coming of AGI is entirely uncertain so we should act now. That's unreasonable in and of itself, and I don't need to go back and read all the history he's got with others to be able to tell that.
Opinion, anyway.
Edit: To reiterate why it's unreasonable- if we won't see AGI until it's already here, what, exactly, are we supposed to be preparing against? We won't know it even when we see it- so how will we know what to protect ourselves from?
Reasoning with uncertainty requires at least some amount of knowledge and the article goes to great lengths to point out there isn't any, in the case of AGI.
So there's no reasoning to be done, either. In that case, what are we talking about? "Beware of the unknown"? Well, OK. I don't know if the sky will fall on my head tomorrow so maybe I should stay home, just in case?
Well, if you have no way to tell whether something is going to happen, or not, you don't prepare for it- because you can't justify spending the resources to prepare. Or rather, in a world of limited resources, you can't prepare for every single event that may or may not happen, no matter how important.
To put it plainly: you don't take your umbrella with you because you don't know whether it will rain or not. You take it because you think it might. Otherwise, everyone would be going around with umbrellas all the time, just because it's impossible to make a completely accurate prediction about the weather and you don't know for sure when it will start raining until the first drops fall.
In the same sense, if there's no way to tell when, or if, AGI will arrive, then it doesn't make any sense to start preparing for it right now. We might as well prepare for an alien invasion. Or for grey goo, or a vacuum metastability event (er, not that you can prepare for the latter...).
In fact, if AGI is going to happen and we can't predict it in time then there's no point in even trying to prepare for it. Either we decide that the risk is too great and stop all AI research right now, or accept the risk and go on as we are.