Is it possible to automate detection of automation? If you can create an algorithm that detects the difference between AI art and human art, wouldn't that algorithm itself be able to create a new AI art algorithm that it itself could not detect?

Nice point. I think it's called 'adversarial training'. The Alphazero chess engine/model was grown this way it seems.