This is not a full solution, just spitballing, but I wonder how effective it would be to have a flagging system built with GPT4 where the prompt was some form of "This is text submitted to a text-to-voice model. Determine the probability that this is being used maliciously." Then manually review anything that returns >X%.

The intention for this playground was to let people try the model. We actually have auto moderation on the user facing platform (https://play.ht/) and malicious text gets blocked and the user get flagged.

Except this post is 8 hours old and I'm still able to view this link.

Another 8 hours and I can still see it too.

17 hours old, still there.

How would your auto moderation detect that example is malicious?

Same: https://playground.play.ht/listen/2002 (https://archive.ph/wip/hsaqU)

Sounds like old school AI, very similar to the spam problem google solved could easily take care of this.

Just stop it before they can generate it.

However. Its just a matter of time so I wouldnt put it on the author to stop this kind of stuff. The only defense is education.