It should be noted that the problems of U+202E and such are not inherent to all Bidi_Control characters: Right-to-Left Marker, Left-to-Right Marker and Arabic Language Marker are all safe (and useful)—though UTR #36 suggests going far enough (e.g. “use only LTR or RTL, and if RTL, first and last characters must be strong”) that they wouldn’t be useful. It’s specific to the characters that override the default stuff, which means Embedding, Isolate and Override characters.

The Trojan Source paper has a decent treatment of the area too, seen in a different application: https://trojansource.codes/trojan-source.pdf.

The problem seems to be that Unicode has gone a bit too far. No need to have the ability to do this.

The need seems quite obvious when mixing LTR and RTL languages, e.g. embed an American company or English phrase in a RTL language text.

Why does it need to overwrite or obscure existing characters?

The Right-To-Left Override (U+202E) does not obscure nor overwrite existing characters. It is an instruction to switch the text direction to right-to-left like for Arabic and Hebrew.

In the article it is shown how this can be misused to show executable files as a different file type:

    file[U+202e]fdp.exe shown as fileexe.pdf
        ~~~~~~~~^^^^^^^
           |       |--- read this right-to-left, i.e. exe.pdf     
           |----------- this is the Unicode character Right-To-Left Override

It certainly obscures and overwrites. When you replace "fdp.exe" by "exe.pdf", you're obscuring and overwriting the leading "f" by "e", then the "d" by "x" and so on.

It doesn't overwrite or obscure, it switches rendering direction. E.g. \u202E\u41\u42\u43 starts outputting from the right: \u41 from the right, then 42, then 43. That's CBA, altough in normal English you'd render that as ABC.

More programming languages/platforms need to include built-in routines for various kinds of unicode sanitization.

Or maybe Unicode shouldn't include this crap in the first place?

If you "sanitize" Unicode you're just removing that functionality. Remove that character, and the right to left reversal wont' be there, right?

If you have to break it in the name of security and sanity, it shouldn't exist in the first place.

The functions/codepoints are there for reasons. If you want to research the reasons and decide that text processing doens't need them, that's one thing. But to simply dismiss it off-hand with the argument "if you have to remove it for security it shouldn't exist in the first place" is just ridiculous. It's like saying "if you have to properly escape parameters in SQL to avoid injection attacks, SQL shouldn't exit in the first place!"

I am continually baffled about why the HN comment sections are such a hotbed of anti-unicode sentiment. Perhaps most of y'all don't have to write software that deals with non-English text in any significant way.

If you just didn't have buffers you wouldn't have to avoid buffer overflow attacks! Come on.

> if you have to properly escape parameters in SQL to avoid injection attacks, SQL shouldn't exit in the first place!"

Yes, this aspect of SQL is poorly conceived because the only way is to manipulate it with text processing.

Parameters should be inserted into a query language using AST manipulation, like a Lisp backquote.

Now within a particular program, we can wrap something like that around SQL generation, and solve the escaping problems in one place.

That approach doesn't apply to character data, which has to be literal and primitive in many situations.

Yes.