Nearly all image generating models can output abusive content. Thats part of the reason huggingface put safety measures in place.

I presume people tried that with chatgpt as there is demand for such content.

Then there’s also the training content. All forums and online platforms have some shape or form of abusive content. Filtering that can be very traumatising because it reveals what some among us think and do. Such content is not rare - it sometimes surfaces on this forum as well but to a lesser extent.

The frequency of such content is a good indication of what prompts people might be coming with.