How on Earth are devs supposed to sanitize this??
By using the widely-available free tools which are near universal by now? This hasn't been technically _hard_ since around the turn of the century[1] but developers, especially Americans, have been lazy and tried to avoid doing the work needed to get beyond C-style string handling or, in the case of the apostrophes, using parameterized queries.
1. Having build a site which supported English, both simplified and traditional Chinese, Thai, and Vietnamese (including what passed for webfonts back then) in the 90s, I know how bad it used to be but also that was 3 decades ago.
Unicode is not a good system. Yes, we can struggle with it but it will always be a struggle because of the nature of its design.
A major omission is a way to indicate the encoding of a particular byte array. You open a file on disk and read its characters -- what encoding is it in? What encoding is used for the various buffers embedded in dozens of historic file formats for audio, video, archives, etc?
Developers are forced to guess and hack. The results are about as expected, given the quality of the underlying interfaces.
UTF-8. Unless you've got clear directions that it needs to be something else, the correct choice is always UTF-8. Stop emitting things in other encodings, "Doctor it hurts when I stab myself with this knife". Yes it does, so don't.
These types of errors are guaranteed specifically because of the policy of guessing and loose heuristics you're describing.
> Unicode is not a good system.
Unicode is excellent. Your gripe is with human languages and their scripts. It wasn't the Unicode consortium who decided that some scripts have oddities like single glyphs that can encode anything between a single sound and a whole sentence, or oddities like different 'cases' where glyphs change form just because they happen to be at the beginning of a sentence or proper noun
Also, what is a "character" to you?
Obviously, because you need to examine characters. Here are a couple examples:
* You're implementing any software that deals with rfc6531 compliant email. You'll need to parse strings as unicode to understand where to route mail, for example to understand 你好@example.com
* You're implementing a database, or any software with a search function which indexes unicode fields. You'll have to understand the logical character structure of the data, or your searches won't match properly.
> Also, what is a "character" to you?
Characters are defined by the unicode specification. In UTF8 they can be represented by multi-byte sequences. Using the example of 你好@example.com, the @ is the seventh byte position, but the third character position. You cannot implement software that handles internationalized emails without parsing and understanding strings like this.
Unicode's glossary offers four definitions for this word, which ought to be a strong hint that this is not a technical term but instead a vague idea people have which doesn't align very well to the technical problem.
This is precisely my point. Internationalization is building on a foundation of sand.
You're much more likely to iterate over elements in a string than index. You might iterate over code points, abstract characters, glyphs, grapheme clusters, etc. You might do it before or after a certain kind of normalisation. Since there are many ways to split a string (depending on what you're doing), it won't be possible to make all of them have O(1) indexing.
Why don't you explain how you would parse the above email address without understanding the unicode encoding?
Naming the smallest unit of information in C a 'char' was a terrible mistake made decades ago. A character has never been a single byte in many countries. Shift-JIS, for example, can be one or two bytes, and it's two and a half decades old. Even in UTF-16 (used by Java and C#), a 'char' isn't actually a character, but a codepoint. One that could even be an unpaired surrogate.
Really, you should never be indexing into the backing byte/char array unless you know what you're doing. If you think you need to, you should be asking yourself, "what is it I really need to do?" Because you, as a programmer dealing with internationalization, must not assume English/Western ideas about language. For example, in abugidas[0], a single word can be a whole sentence. For a more common language, Japanese typically doesn't use spaces because they're not needed when writing kanji. But that's not foolproof because hiragana and katakana aren't for words, but sounds. How do you do a word count in those languages? You don't.
> you should use a URL library, not some hack
I think what you're somehow missing is that I am specifically offering authoring an email parsing library (or URLs, or whatever) as an example of where things break down.
Your answer is essentially "let someone else program it, because it's complicated and fraught with peril" -- but my entire point is that it's complicated and fraught with peril.
You are agreeing with me, and your post is a fine example of why the interfaces are bad and error prone.
Are you asking how one would write code for the libraries you should use? In general, there already is one: ICU.[0] But if you had to write your own, as I said above, Unicode provides algorithms[1] that one might want to implement, in addition to a massive (over 1000 page) spec[2] detailing how to handle the various languages. For example, if you need to write the code to handle bidirectional text, there's UAX/TR #9[3]. Want to implement case changing? See Chapter 3.[4]
There's also the Unicode Character Database[5] that contains information on every code point. Your library/program would need to hold a copy of this whole thing.
[1]: https://unicode.org/reports/
[2]: https://www.unicode.org/versions/Unicode15.0.0/
[3]: https://unicode.org/reports/tr9/
[4]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf
You said that people ought never address text by characters or bytes.
I'm pointing out that this is nonsense and I offered several examples.
You responded saying basically that those tasks are complex and someone else should do it.
In your followup response, you recommend using ICU which ... has an interface for iterating by character. Which is all I had mentioned to begin with. Sigh.
Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken or you’re writing an implementation from scratch. In that case, treat it as an opaque blob.
We are talking about email software, inclusive of servers, clients, plugins, etc. It's literally the example I'm offering (but there are neigh infinite others)
> Send the email and have the user validate it. I
But I haven't proposed a problem of validating an email. Besides, even if the problem is validating an email this is not necessarily a workable answer.
> Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken
A lot of the arguments supporting unicode seem to boil down to acknowledging that all these things are broken, yet disregarding them because the problems exist at a more technical level than most of HN commenters are familiar with.
You might not write this kind of software for a living, but some of us do and we know that these interfaces are not good.
Even ASCII has upper and lower case letters. Languages are hard.
That said, you could clearly whitelist a much more reasonable range of characters than just the Latin alphabet. Hypens, apostrophes, common accented letters etc. should probably all be allowed. There are standard algorithms for canonicalising the unicode representation of them.
For transform operations like "toLower" often it turns out that to the extent it means anything it's only relevant to ASCII and so an operation which only transforms ASCII is appropriate anyway.
The list is infinite, but let's use a very simple one which is the focus of this article: Names of things.
You name a file on a case insensitive filesystem. Should the case insensitivity only apply to certain encodings, or all of them? You'll need to do a case transformation and string comparison to make this work.
You create an email address, per rfc6531. You'll need to deal with case flattening, string comparisons, and lexical parsing.
These are not strange operations. These are not only common they are obviously required operations, because names identify things and names (eg emails) are often structured and must be parsed and understood.
Can you imagine the fallout if software developers said "sorry, you can only search for ASCII names because we think indexing unicode strings is a very strange thing to do?"
Better to say you need a canonicalisation table and you will need to canonicalise the text. NTFS filesystems actually carry their canonicalisation table with them, so if you happen to be using a USB drive with an NTFS filesystem where Z and Q are just different versions of the same hypothetical "Q-Z" character, filenames ZZ150 and QQ150 are indistinguishable on that USB drive, even though your Windows PC has no trouble distinguishing Z from Q.
Notice that unlike "case transformation" this doesn't pretend we're doing something with larger semantic value, and indeed the actual names on disk aren't promised to be Unicode text under NTFS, they're just arbitrary 16-bit values which probably make sense as UCS-2 or these days UTF-16 when rendered, the tables just convert some values into others.
How do you propose that I, a person who among other things develops filesystems, implement a canonicalization table without doing these "very strange things?"
Furthermore, filesystem operations are not limited to calls into the filesystem driver. Programs often need to deal with paths outside of filesystem calls: The names of files need to be displayed on screens. They need to be indexed in databases and searched for. They need to be encapsulated within other protocols and data formats and often need to be parsed and understood.
You are essentially agreeing with me that these are normal, necessary things. Your position appears to be "someone else does that type of thing for me," which is not a very compelling argument.
It's just a particularly common canonicalization function. Canonicalization is a common need across all languages as we've already clearly established.
I can't find where rfc6531 requires case flattening. The word "case" is used there in the meaning of "situation".
> Check if file exists or generate numeric name.
Nonsense. You may not be in control of the names, and checking a file with stat(2) may not address the problem at hand (and: is many orders of magnitude different in cost as well).
Or possibly ISO-8859-1, which is, helpfully, slightly different.
[1] https://en.wikipedia.org/wiki/Scriptio_continua
[2] The bullet • (U+2022) alt code is 7, or Opt + 8 on Mac. The interpunct · (U+00B7) alt code is 250, or Shift + Opt + 9 on Mac. Rather inconvenient for the regular user.
I would love & natively though. For my wedding (bride&groom.com for example) I looked into faking it with punycode but nothing was close enough, and anything other than an ampersand would have been too hard to input.
You just can't have them in domain names.
Aside from all the comments about what's wrong with this on a technical level, this is a poor example since the product name is actually "Coca-Cola" and not "Coca Cola".
Then there’s the issue of character encodings, which the article does a decent job explaining! Nitpick though: they claims it’s a UTF-8/ASCII thing, but it’s actually a UTF-8/Windows-1252 issue.
In related news, even GNU coreutils fails to support UTF-8 properly despite a claim of support for multibyte character sets: https://catgirl.ai/log/cut-c-harmful/