You either die an MVP or live long enough to build content moderation

Re:You either die an MVP or live long enough to build conten... by jffry on Tuesday September 28, 2021 @01:25PM (#28685229)

In fact, the Unicode consortium provides a report and extensive list of "confusable" symbols, which you could use alongside Unicode normalization tables to map adversarial back into more ASCII-equivalent text before running it through anti-spam mechanisms that are interested in the content of the message.

https://www.unicode.org/reports/tr39/

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by wanderingstan on Tuesday September 28, 2021 @01:36PM (#28685341)

I built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO.

https://github.com/wanderingstan/Confusables

E.g. "𝓗℮𝐥1೦" would match "Hello"

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by Alir3z4 on Tuesday September 28, 2021 @01:40PM (#28685370)

I should have had better eyes while searching, could find this and saved some hours.

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by jffry on Tuesday September 28, 2021 @02:01PM (#28685602)

I only learned about it myself after spending too long building my own half-baked version. I think it's in pretty opaque language that makes it hard to find even if you know what you want.

Maybe somebody else on here will see it and learn about it before they need it, and at least you still have a new tool to reach for in the future.

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by mr_toad on Tuesday September 28, 2021 @08:49PM (#28689154)

What I don’t understand is who sees spam that looks like an old school ransom note and thinks ‘seems legit’, and then actually buys something?

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by airstrike on Tuesday September 28, 2021 @02:08PM (#28685680)

You should post this to Show HN. Also you have a typo on your README ("characgters")

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by jart on Tuesday September 28, 2021 @03:38PM (#28686564)

The one built-in to Python will get you most of the way there:

    >>> import unicodedata
    >>> unicodedata.normalize('NFKD', '𝓗℮𝐥1೦𝗵𝗲𝗹𝗹𝗼')
    'H℮l1೦hello'

Obviously it isn't going to remap leetspeak characters like 1 -> l but it covers a lot of cases.

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by zerocrates on Wednesday September 29, 2021 @12:40AM (#28690372)

Obviously you're saying it doesn't cover everything, but a big thing it's not going to catch beyond leetspeak-type situations is the kinds of thing you (used to) see in internationalized domain spoofing: legitimate non-Latin-script letters that just look the same or nearly the same.

NFKC/NFKD will handle "this is another form of the Latin letter A" type stuff but not "Cyrillic A looks like Latin A."

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by wanderingstan on Tuesday September 28, 2021 @02:19PM (#28685809)

Thanks, I've fixed the typo! It was such a simple project, hardly seems worthy of a "Show HN".

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by jonplackett on Tuesday September 28, 2021 @02:48PM (#28686078)

I've seen crazier things get to #1

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by 8note on Tuesday September 28, 2021 @03:03PM (#28686196)

Test for the library: would it catch that that typo still refers to characters?

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by JanisL on Tuesday September 28, 2021 @07:39PM (#28688720)

Thank you for making this!

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by jonplackett on Tuesday September 28, 2021 @02:46PM (#28686063)

There should be a 'bad symbols' list you can block like you do expletives. There's surely zero need to support that kind of thing in comments.

[ Parent | on HN ]

Re:You either die an MVP or live long enough to build conten... by jcims on Tuesday September 28, 2021 @07:18PM (#28688581)

One term used to describe this is homoglyph, as in homoglyph attacks for phishing.

Back in 2015 I did some work just using simple bitmap rendering plus OCR to find text that looked like a small selection of known words. It was actually reasonably effective.

[ Parent | on HN ]