In fact, the Unicode consortium provides a report and extensive list of "confusable" symbols, which you could use alongside Unicode normalization tables to map adversarial back into more ASCII-equivalent text before running it through anti-spam mechanisms that are interested in the content of the message.

https://www.unicode.org/reports/tr39/

I built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO.

https://github.com/wanderingstan/Confusables

E.g. "𝓗℮𝐥1೦" would match "Hello"

I should have had better eyes while searching, could find this and saved some hours.

I only learned about it myself after spending too long building my own half-baked version. I think it's in pretty opaque language that makes it hard to find even if you know what you want.

Maybe somebody else on here will see it and learn about it before they need it, and at least you still have a new tool to reach for in the future.

What I don’t understand is who sees spam that looks like an old school ransom note and thinks ‘seems legit’, and then actually buys something?

You should post this to Show HN. Also you have a typo on your README ("characgters")

The one built-in to Python will get you most of the way there:

    >>> import unicodedata
    >>> unicodedata.normalize('NFKD', '𝓗℮𝐥1೦𝗵𝗲𝗹𝗹𝗼')
    'H℮l1೦hello'

Obviously it isn't going to remap leetspeak characters like 1 -> l but it covers a lot of cases.

Obviously you're saying it doesn't cover everything, but a big thing it's not going to catch beyond leetspeak-type situations is the kinds of thing you (used to) see in internationalized domain spoofing: legitimate non-Latin-script letters that just look the same or nearly the same.

NFKC/NFKD will handle "this is another form of the Latin letter A" type stuff but not "Cyrillic A looks like Latin A."

Thanks, I've fixed the typo! It was such a simple project, hardly seems worthy of a "Show HN".

I've seen crazier things get to #1

Test for the library: would it catch that that typo still refers to characters?

Thank you for making this!

There should be a 'bad symbols' list you can block like you do expletives. There's surely zero need to support that kind of thing in comments.

One term used to describe this is homoglyph, as in homoglyph attacks for phishing.

Back in 2015 I did some work just using simple bitmap rendering plus OCR to find text that looked like a small selection of known words. It was actually reasonably effective.

Those are known as homoglyphs. This issue has been studied and best practices are documented. There's no complete solution but it can be mitigated.

https://en.wikipedia.org/wiki/Homoglyph

All kinds of possibilities there.

Unicode Character “;” (U+037E) is a beaut.

https://news.ycombinator.com/item?id=28621681

Yeah, then someone has to create or find that whole table and make.

The initial problem wasn't those symbols but the content itself, the symbols and special characters came into the problem later.

Later on as mentioned in my original comment, that they would use positive content from other blog posts that were published/passed the moderation to mix up their bad content.

Probably could use a different method, but at that time needed something quick and fast and it worked and still works with very little tweaking.

Although we don't have massive amount of threats or abusers anymore to exactly know the effect, but again, so far it works.

That time, they would coming several thousands per minute, IP blocking, range blocking, USER AGENT, captcha or anything such didn't work on them.

The good news is that the Unicode consortium has a report on this issue, and the tables already exist for normalization and mapping of confusables to their ASCII lookalikes: https://www.unicode.org/reports/tr39/

Oh, that's nice.

I guess I can use that next time time to work on the data cleaning for that model.

Thanks.

As I commented above:

I built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO. https://github.com/wanderingstan/Confusables E.g. "𝓗℮𝐥1೦" would match "Hello"

If you can identify text written with mixed glyphs just ban it outright. Normal users don't use text like this, the pure binary presence of such "homomorphic" text at all is probably a better signal for spam than whatever your neural net when running it after normalization.

> Normal users don't use text like this

I think that depends on the users. People copying and pasting bits of text that was in English or another common language— think documentation, code, news articles, tweets, etc.— with a different character set could be problematic.

Also, 𝒮ℴ𝓂ℯ 𝒜𝓅𝓅𝓈 marketed as "𝔽𝕠𝕟𝕥𝕤 𝕗𝕠𝕣 𝕤𝕠𝕔𝕒𝕝 𝕞𝕖𝕕𝕚𝕒" would be ℭ𝔞𝔲𝔤𝔥𝔱 𝔲𝔭 𝔦𝔫 𝔱𝔥𝔦𝔰. (math symbols) A user base with young people getting bounced or shadow banned for trying to express themselves or distinguish themselves from their peers would be like ಠ_ಠ (Kannada letter ttha)

I think targeting the language they're using is a better bet.

¯\_(ツ)_/¯ (Hirigana letter tsu)

I can especially echo the "social media fonts" trend. They're quite popular on certain Discord guilds at least.

Minor nitpick, but ツ is the katakana tsu.

Oh, fact. Not a Japanese speaker. (or reader)

> pasting bits of text that was in English or another common language

If they use many (maybe three? four? or more) character sets in the same post, or different character sets in any single word, then that'd be highly suspicious?

Whilst still letting people copy paste from another language

Special case needed for the shoulder shrug with an Hirigana letter tsu I mean katakana tsu

I've noticed much more usage of alternative Unicode ranges for numbers/letters in email subjects lately to make marketing messages stand out, too (in addition to emoji of course), though I wouldn't necessarily mind banning that...

Bizarrely, I'm seeing recruiters using this on LinkedIn.

huh. For any specific purpose? Does it seem like they avoiding paying for recruiter accounts or something by evading algorithms designed to detect their activity, or is it just for the heck of it?

It's following some business wank advice to stand out from the crowd like printing your resume on A3 card stock.

Ah. Sounds like something one of those pickup artists would say if they blindly pivoted into marketing.

> Normal users don't use text like this

Sounds like you live in a filter bubble.

(╯°□°)╯︵ ┻━┻

(ﾉ◕ヮ◕)ﾉ*:･ﾟ

I know right? There are so many times when I've wanted to use something like box drawing unicode characters (cp437) to explain a complicated concept on hacker news, but alas I couldn't, due widespread computer fraud and abuse. How are we going to build a more inclusive internet that serves the interests all ALL people around the world, regardless of native language, if the bad guys are forcing administrators to ban unicode? (╯°□°)╯︵ ̲┻̲━̲┻

> Normal users don't use text like this

They kinda do. Check out the shrug "emoji", table flip, and so forth. Then there's the meme of adding text above and below by abusing Unicode's "super" and "sub" modifications.

You could block it to only ever represent ASCII, but then you've knocked out the ability to expand internationally.

You can hardcode a rule for this specific bypass. Or you just retrain the neural net, and it learns that presence of these symbols = bad very quickly, and you spent less time writing and testing a custom solution.