Re:Scunthorpe Sans – a self-censoring font
by tzs on Sunday May 10, 2020 @08:43PM
(#23137742)
I had occasion to write a chat server for a small gaming service once. The way I came up with to do the profanity filter ended up working a lot better than I expected it to, and avoided the problem you describe.
It worked like this.
1. Make a copy of the text to be filtered, modified as follows.
-- change upper case letters to lower case
-- change 0 to o, 1 to i, 3 to e, 5 to s, and vowels with various accent marks to the corresponding unaccented lower case vowel
-- discard anything else
For example, if the input was "Start the assault. let's f-u-c-k them" it would become "starttheassaultletsfuckthem".
2. Scan this for any substrings that are on the bad word list. In this case it would find "ass" and "fuck" [1].
3. If no bad words are found, the original string is returned and we are done.
4. For each word in the original string, look it up in the dictionary (I believe I used /usr/dict/words). Note the ranges of character positions in the original string of the words that are spelled correctly.
5. For each bad word found in the filtered string, if all of its characters came from positions in the original string that were part of correctly spelled words, leave it alone. Otherwise, asterisk out its characters in the original string.
In the above example, "ass" would be left alone because all of its characters came from "assualt" which is recognized as a correctly spelled word. None of "fuck" came from correctly spelled words, so it would get zapped. The final result would be
start the assault. let's *-*-*-* them.
[1] Well...actually not. I just checked my archives and found the chat server bad word list. It did not include "ass".
It worked like this.
1. Make a copy of the text to be filtered, modified as follows.
-- change upper case letters to lower case
-- change 0 to o, 1 to i, 3 to e, 5 to s, and vowels with various accent marks to the corresponding unaccented lower case vowel
-- discard anything else
For example, if the input was "Start the assault. let's f-u-c-k them" it would become "starttheassaultletsfuckthem".
2. Scan this for any substrings that are on the bad word list. In this case it would find "ass" and "fuck" [1].
3. If no bad words are found, the original string is returned and we are done.
4. For each word in the original string, look it up in the dictionary (I believe I used /usr/dict/words). Note the ranges of character positions in the original string of the words that are spelled correctly.
5. For each bad word found in the filtered string, if all of its characters came from positions in the original string that were part of correctly spelled words, leave it alone. Otherwise, asterisk out its characters in the original string.
In the above example, "ass" would be left alone because all of its characters came from "assualt" which is recognized as a correctly spelled word. None of "fuck" came from correctly spelled words, so it would get zapped. The final result would be
[1] Well...actually not. I just checked my archives and found the chat server bad word list. It did not include "ass".