This assumes 'we' (ie societies) are in a position to stop it - whether that's nuclear weapons or AI. If we are not, then what can be usefully done is going to shift… by a lot.

> This assumes 'we' (ie societies) are in a position to stop it

There’s major advantages to understanding the world as it is independent anything else. People make tradeoffs around harm all the time, pretending it doesn’t exist is pointless.

We can mitigate harm from earthquakes and blizzards independently from our ability to prevent such events. That comes from understanding such events as more than just acts of gods who would happily use other means should we try to mitigate the harm from earthquakes etc.

We might want to treat two things differently when for one of them, its only function is unimaginably massive destruction and for the other it’s to produce words and images.

Treating them differently based on the harm they cause, is still judging them based on the harm they cause rather than treating them as a neutral entity.

I don’t think it makes any sense to ignore the immediate consequences of using/abusing a tool when trying to determine the nature of any regulations or other curbs around that tool.