They did consult not just experienced East Asian programmers, but Chinese, Korean, Japanese and Vietnamese professors of Linguistics and the relevant local authorities, and had them form the Ideographic Rapporteur Group, which advised them that all the East Asian languages can be encoded in 20940 code points.
Unicode is not centrally run like so many people seem to expect. Rather, the Unicode Consortium itself acts just as the standards body that mates together local standards into a single complete whole. The Consortium makes no decisions about how Chinese is represented, other than allocating them code pages; all the relevant decisions about any language is done by local experts. The problem was that the early 90's were an era of boundless optimism and futurism, which in East Asia resulted in the Han Unification project, with the idea of cutting down the amount of symbols in use, and unifying the representation of all the languages that used a Han-derived alphabet. That... did not go so well.
To my eyes, the letter "A" in English (Latin), Greek, and Russian (Cyrillic) looks identical. Why does it need three separate code points? Maybe it gets tricky when the relationship between uppercase and lowercase diverges. In fact, that does appear to be one of the arguments in this technical note:
>Even more significantly, from the point of view of the problem of character encoding for digital textual representation in information technology, the preexisting identification of Latin, Greek, and Cyrillic as distinct scripts was carried over into character encoding, from the very earliest instances of such encodings. Once ASCII and EBCDIC were expanded to start incorporating Greek or Cyrillic letters, all significant instances of such encodings included a basic Latin (ASCII or otherwise) set and a full set of letters for Greek or a full set of letters for Cyrillic. Precedent for the purposes of character encoding was clearly established by those early 8-bit charsets.
That's true, although it reminded me of a cool charset: the Russian KOI8-R encoding[1]. This encoding was created so that if you stripped the high bit (presumably on a system unable to handle russian properly) you ended up with semi-legible latinized Russian. Quoting wikipedia:
>For instance, "Русский Текст" in KOI8-R becomes rUSSKIJ tEKST ("Russian Text") if the 8th bit is stripped; attempting to interpret the ASCII string rUSSKIJ tEKST as KOI7 yields "РУССКИЙ ТЕКСТ". KOI8 was based on Russian Morse code, which was created from Latin Morse code based on sound similarities, and which has the same connection to the Latin Morse codes for A-Z as KOI8 has with ASCII.
This legacy support is why there are precomposed characters.
It’s not quite the same for Han unification (people using those alphabets were already used to complex gyration was ao e). In that case the linguists, historians and other scientists were in agreement but they neglected politics. In Latin unification nobody cares that J sounds different in every language, but because Hanzi embody semantics, Japan may have a loancharacter because it once sounded the same as a word in use at the time OR because it had the same meaning as a word in use at the time so the sense of “unified” can be contentious, even though they are simply code points.
Sorting complex charsets may require some sort of collation, of course, but in simple cases the status quo is highly usable without one.
1. Let's start with a simple character (indeed the character for character) like 字. The top stroke on this character is generally written straight in Japanese (and I believe Korean), but it has a very distinct diagonal angle (top left to bottom right) in Chinese, both Traditional and Simplified. Should it be counted as the same character? I believe most linguists would say yes. If your answer for this is no, then Han Unification is a lost cause - you'd basically have to create completely separate character sets for Simplified Chinese, Traditional Chinese, Japanese and Korean and Vietnamese, and add a few extra characters to the Traditional Chinese set that are different over in Hong Kong.
2. Let's go further on with the grass radical (艹) which has 4 strokes in Traditional Chinese vs. 3 in other writing systems. Should all the characters with this radical be separate?
3. What about characters like 骨? The inner right angle stroke in the top of this character is on the left side in Simplified Chinese, and on the right side otherwise. Similar minor differences can be found for characters like 刃 and 化. These are not really the fault of simplification, but rather come from the fact that different countries chose a different Han character variant to standardize on.
A good example from another non-western language that went through a partial unification ignoring stylistic differences in writing would be Urdu. It's sharing the same unicode range as Arabic, even though the typical styling of "the same characters" in both languages could be quite different. Here's a signboard with both Urdu and Arabic from Wikipedia - the difference is quite apparent: https://upload.wikimedia.org/wikipedia/commons/2/22/UAE_sign...
But the unification could never be perfect. Simplified characters often combine more than 1 traditional characters. e.g. 发 could mean both hair (髮) and discharge (發). Japanese simplified the later character to 発. Unicode assigns a different codepoint to each of them, and it would obviously be quite bad for Traditional Chinese if it didn't. It also assigns a different code point where consistent stylistic simplification of radicals are involved, so all characters with the radical 見 ('see') have different codepoints from those with the simplified radical 见.
Even ignoring political tensions, Han unification was naive from the start, especially considering they knew they wouldn't be able to unify everything. They could have left a way to designate regional styling within the encoding.
The reasoning is not totally crazy; the Table of General Standard Chinese Characters even today only has about 8000 characters.
You can't leave out people who need to encode rare languages on computers underserved, so for all the scholars recording Egyptian and Luwian hieroglyphs, Sumerian and Akkadian cuneiform and Tangut script. You'd probably have to eventually support as-of-yet unencoded scripts such as Classical Yi, Jurchen, Aztec hieroglyphs and so on.
You'd have to add alphabets and syllabaries for small minority languages such as Lisu, Coptic, Cherokee and Vai. And you'd have to support the rather widely used writing systems of India, each with its own 128 characters.
Combine this with the over 28,000 CJK characters Unicode did end up allocating, the 11,184 discrete codepoints they Unicode did end up adding for Koran Hangul [1], you've already used 40,000 characters. Add all the long-tail scripts and you're past 65,535 well before we even reach Emoji or extremely rare CJKV characters from Mojikyo[1].
"In other words, given that the limitation to 65,536 character codes genuinely does satisfy all the world's modern communication needs with a safety factor of about four, then one can decide up-front that preserving a pure 16-bit architecture has a higher design priority than publicly encoding every extinct or obscure character form. Then the sufficiency of 16 bits for the writing technology of the future becomes a matter of our active intention, rather than passive victimization by writing systems of the past."
The engineers behind Unicode made the classic software engineering mistake of looking at their needs today and taking a factor of safety that's way too small. It's the same thing that happened with the IPv4 address range. A simple back-of-the-envelope calculation would have told us it's not going to be enough, but it was easy to forget back in the 1980s that one day children in Africa would also have personal computers connected to the grand Internet, that would need an IP address.
The fact that by version 1.0.1 (the first version of Unicode to include the CJK characters), they already used almost half of their space when they were planning to use less than a quarter, should have already prepared them to the future. No, the reasoning behind 16 bits was always crazy. You can't design future-proof standards with a low safety factor and condescending estimation of usage (Yi Syllabary? Who would ever use that?).
But we should probably be happy about this. A 32-bit encoding would have certainly been rejected by just about everyone as too wasteful, and software engineers back in the 80's and 90's were still unable to give up merging their understanding of codepoints with glyhs, so a multi-byte encoding (which is what we ended up having eventually with UTF-8 and UTF-16) wasn't a popular idea either.
Fortunately enough, Unicode is still much better than the mess it preceded, and the fix for the 16-bit limitation, as painful as it was, has proven far easier than IPv6.
[1] a.k.a The "Korean Mess": http://unicode.org/pipermail/unicode/2015-June/002038.html
They are not 16 bits wide, but mostly 14 bits wide and separated from the single byte portion with most significant bits of bytes set. As far as I can tell there are "only" about 33,000 code points [1] in the Unihan database that are known to be officially encoded in some sort of encodings throughout the entire East Asia. This does not include unofficial but de facto encodings like ETEN extensions to Big5 but you get the idea. This and the Han unification made it pretty reasonable to expect that the modern characters can be encoded in 16 bits.
[1] I've counted distinct code points in the "other mappings" table from the Unihan database.
That's not what Unicode does though. Unicode encodes many duplicate characters (because they are duplicated in a national encoding standard and the round-trip compatibility requirement). Unicode also includes every possible variant that is of cultural or academic interest, including dead languages and so-called "ghost characters" that are actually printing errors in old paper dictionaries or census data, but now allocated a code point. And this is before we say anything about emoji...
So from today's perspective 16 bit encoding space is laughably small. But it is true that basically any message meant to be consumed by the general population, including CJKV languages but notably not emoji, can be fully represented with characters found on the BMP, which can be faithfully encoded in UCS-2.