However, this is only the case if you have just a big blob of text and UTF-8 might be better if the east asian text is interspersed with ASCII characters. For example, HTML with east asian text often is smaller when encoded with UTF-8, because a significant portion of the content is HTML markup.

Right well again, if your average East Asian codepoint is 3 bytes in UTF-8, and your average markup codepoint is one byte, then as the percentage of markup in your corpus rises you'll grow asymptotically towards 1. But not all text contains markup ;) Consider any database storage for example, text in a game, e-books, visual novels. I guess anything that isn't the web or JSON is what I'm saying -- which is a lot.

Also bear in mind that gzip (or any non-toy compression algorithm) will do much better than either on any non-contrived input.

Also on UTF-16:

> Alas, if your language needs to run on or interoperate with the browser, the JVM, or the CLR, you might be stuck with [UTF-16]

And Windows. Win32 and COM APIs use UTF-16 natively, so if you want to avoid lots of conversions on Windows, UTF-16 is the way to go. Maybe the best solution would be an abstraction that lets you choose the internal string encoding at compile time.

Hmm, interesting. I guess they'd likely end up three bytes in UTF-8. Do you have any details you can point me at?

Nothing like a paper or anything, but just kind of anecdotally I downloaded 西遊記 from Project Gutenberg [1] and measured it. In UTF-8 it's 2,236,564 bytes, in UTF-16 it's 1,554,998 bytes.

[1] https://www.gutenberg.org/files/23962/23962-0.txt