I run an online community, with my own developed software. We use lots of different tools to prevent spam, harassment, etc. We use different APIS to check IP, ASN, email for previous spam behaviour, use of disposable email domains, etc.

We have a team of moderators and as mentioned before you can "smell" something brewing. The words, to whom they are directed, etc.

A couple of years ago we started using Perspective API [0], a Google service "Using machine learning to reduce toxicity online". We use the toxicity model, trained by Google and The NY Times. Basically we submit the text of posts and replies and receive a score. We do some account history calculations and if it's past a certain threshold then moderators receive a notification - we can't read all posts in our platform all the time but this helps a lot.

The community itself helps too, and the reporting tool is used when things happen that we aren't aware.

[0] https://www.perspectiveapi.com/#/home

There are a lot of projects out there that are attempting to do this because of the impracticability of a human team to deal with the entire torrent of information.

For example https://playvalorant.com/en-us/news/announcements/valorant-v...

> As part of a larger effort to combat disruptive behavior, Riot Games recently updated its Privacy Notice and Terms of Service to allow us to record and evaluate in-game voice communications when a report for that type of behavior is submitted—with the goal of kicking this off in VALORANT first.

I believe it would be quite possible. Text is a bit dryer to work with but has the advantage of being all text.

I'm pretty sure that's exactly what FB and Twitter are trying to do.

Not sure how well it's working out.

I hear constant stories about how "blunt instrument" these are, and how it's impossible to appeal.