The biggest short term harm comes from their utility, anything that enables an individual to do something that previously required a group stands a greater chance of a single insane/radical/extremest finding a way to do something terrible on their own when they couldn't previously. The oft cited example is someone developing a biological weapon with AI assistance. While you could say we already have laws saying you can't do this, that offers little protection against the scenario where the party performing the action is undetected until it is too late.

I see some AI regulation proposals specifically prohibit AIs that might assist to biological weapons. This strikes me as missing the point. The risk isn't something we have thought of but AI enabling something catastrophic that we haven't thought of.

That has the exact same energy as "encryption could be used for nefarious purposes, like sedition or CP" therefore we need to regulate encryption.

We wouldn't want to enable the bad guys, right?!

You might be misreading it.

It not that it can be used for nefarious purposes, it's that it might render a catastrophic situation vastly more likely.

There are lots of nefarious uses for AI that shouldn't be regulated specifically at the AI level. Generating an image with an intent to mislead could be done with AI, but it could also be done in Photoshop(often better). AI could make it more efficient as part of it making things more efficient in general. That sort of thing should be addressed at the level of existing laws, the bad part is not intrinsic to the AI.

Like some sort of lab conducting gain of function research on contagious diseases?

That does scare me tbh.

Anything involving life or death decisions

Regulations will go something like this: 1) anything that can be harmful, say targeting of a population, isn't allowed to be owned or be accessed-available for the individual, 2) except for government and state-funded [bad] actors who have a "legal" monopoly of violence - those governments who use that usually captured/corrupted and of authoritarian-tyrannical nature.