Why should "we" not legislate that any AI systems must identify themselves as such when asked? There could even be a specified way to ask this question so it can be recognised by simple NLP techniques and avoid the black box processing of the model itself. This could carry legal weight.

That way, humans could impersonate AIs, but AIs would be legally encouraged, shall we say, not to impersonate humans.

"It could never be enforced" or "but there will be bad actors who don't do this" are useful and valid discussions to have, but I think separate to the question of if this would be a worthwhile regulatory concept to explore.