Being safety first means others with more nefarious values may achieve it first. AGI will eventually arrive at the same singularity point, but between initial creation and that point, it could be extremely destructive. What would happen if authoritarians authored it first?

Would if have been better to go safety first with nuclear weapons so that Germany might develop them first in WW2?

I find that by going very fast and in the open, humanity has the best chance of producing safe AI. Slowing down and making it closed will result in a much worse outcome. Who gets to make the decision on who gets to utilize that kind of knowledge power? Why is such a small group of people being given power over all of the collective human sacrifices and knowledge to get to where we all are?

If we’d done that with the Internet we’d still have dialup while we debate the safety of higher speeds and the danger of viruses spreading faster or whatever.

I’m not arguing that safety is nonsense. I’m arguing that humanity doesn’t know how to both move and be safe, at least at scale. We can do it as individuals when we exercise caution in an endeavor, but in groups we either move or we do not move. When we admit the cautionary forces it becomes a vetocracy of fear and everything stops.

This is why it usually takes an unhinged maniac to make progress at anything, like the psycho behind the company launching the rocket today.

We are not good at this.

Be careful with reasoning by analogy and mistaking what is basically an ideology, namely the progress ideology, for some kind of natural law.

There must be some middle ground between reckless advancement without safeguards and hamstringing the technology through an abundance of caution.

There is, but we are absolutely terrible at any middle ground at scale. People stake out extreme positions, like Yudkowski vs e/acc, and defend them to the death.

Nuanced thinking doesn’t fit in a short memetic sound bite or 140 characters.

Given that we are awful at moderation and if I am forced to choose, I say go. As it stands our technological society will not survive without major advancements in areas like energy, and AI could help us do that.

Contrast that with the risks which are far more hypothetical than nuclear power. As it stands zero people have been harmed by an AI without a human instructing the AI to do harm. There have been no “meltdowns” that were unintended and we are not even sure it’s likely or even plausible.

Our civilization is like a transatlantic flight that has passed the half way mark and no longer has the fuel to turn around. The mechanism here is that our population is vastly higher than what can be supported without technology. The evolutionary ratchet has clicked. We are committed.