This is Google. It's probably very smart pattern matching for profanity.

The neural network may have taken millions of core-hours to learn to be as dumb (here) as a blind keyword search.

We had to give up our privacy to create a highly sophisticated technology that doesn't even work half of the time. I love the future, it was totally worth it.

Well, obviously. If it were a dumb profanity filter then it would be possible to fix it!

I once had a bug that I traced back to a rule (can’t remember in which part of the stack - though I think it was client controlled IIS) that was striping the “select” from the word “selected” in query string params in an attempt to thwart sql injection. From memory it was naive enough that “sselectelect” was converted nicely in the process.

Similar: Yahoo used to (2002) replace any instance of the character sequence 'eval' (and other 'bad' strings) in their emails, in an attempt to prevent Javascript exploits. Needless to say it created a small amount of havoc!

http://news.bbc.co.uk/1/hi/sci/tech/2138014.stm

https://en.wiktionary.org/wiki/medireview

I hadn't heard of this and I'm now flabbergasted. Is it even legal for a service provider to secretly change email contents? It's absolutely outlandish to imagine how someone first thought this could be a good idea, and then found someone capable of executing the plan and apparently agree.

I had similar issues, the software is Mod Security that some hosting companies use and some rules will empty out your POST request field if it contained text like ".... select ...from..." where the 2 keywords were paragraphs apart.

Not super relevant or anything but I just can't help but share my favorite profanity filter story, so here you go.

I worked at a place that had a profanity filter in two parts.

The first part was in C, several pages of if (!strcmp(x, a)) return 0;

After all that, it then invokes popen() to ssh to another machine and run a shell script there, which contains several more pages of string comparisons, this time in shell.

Doesn't popen() pass strings to a shell? Sounds dangerous, as you would have to escape semicolons, quotes, etc.

I might be wrong but I think it's about censoring the 'hell' in 'shell'. Because some parts of the world consider words like 'hell' and 'damn' to be profane.

Hold on, what? Okay, return false if they aren't equal, then open another process to repeat this method once again in the shell... I can't guess the reason. Would you know if there is any reason this might have been done?

I wouldn't know the real reason for sure, but this seems plausible:

1) They got tired of having to modify C code and wait for the deploy cycle to modify the filter

2) Using, for example, the database would be more work than calling a shell script. On top of that, it might actually be beyond the abilities of the programmer involved.

3) The C code executes on an arbitrary machine. Hence the ssh to a specific machine, so that the shell script would only have to be maintained in one place

strcmp returns 0 if the strings are equal.

A great many places do this and automatically refuse content based on arbitrary “bad words” regardless the context.

I remember being denied to post a forum post containing the phrase “tardive dyskinesia”, as it appears that it rejected anything with the string “tard” in it.

I'm not sure as to whom they think to be helping with that, but it's entirely possible that their advertisement revenue will actually suffer, if the string “tard” be found on their pages.

FWIW, general profanity detection is a highly nontrivial problem. It’s true that such subword profanity filters aren’t that great, but slightly more sophisticated ones (eg whole word matching or n-grams) tend to have relatively good precision. You could train a fancy neural network, but the overall return on precision and recall tends to be not that great (compared to the exponential change in speed and cost). The problem almost always crops up in out-of-distribution sentences (such as “bone” at a paleontology conference).

Even humans with full general intelligence and domain knowledge will fail at profanity detection. I think the problem here is not so much that there are false triggers, but that there is no way to deal with the false triggers — no way to appeal to reason or utility.

It's a problem with a subjective answer.

One man's profanity is not another man's profanity.

Of course, the personality trait of desiring censoring “bad words” seems to highly correlate with a belief in objective morality. — the others are wrong about what they find profane!

They just rebranded it as "AI-powered profanity filter" :)